Approach for processing audio data at network sites

Several approaches are provided for processing audio data to generate transcription data that is supplemented with visual content items. The visual content items may be any type of data that may vary depending upon a particular implementation. Examples of visual content items include, without limitation, images, videos, symbols, etc. Embodiments include adding visual content items to transcription data based upon user input, specialized keywords contained in the transcription data and various correspondences with the audio data, including time-based correspondence and correspondences based upon a common user, storage location or logical entity.

FIELD OF THE INVENTION

Embodiments relate generally to processing audio data at network sites. SUGGESTED GROUP ART UNIT: 2625; SUGGESTED CLASSIFICATION: 358.

BACKGROUND

Audio recording capabilities are now available in many different types of electronic devices. For example, audio recording capabilities are available in small, portable, stand-alone audio recording devices, and also in many types of consumer electronic and telephony devices, such as smartphones. The portability of these devices allows users to easily record audio from almost any location. In addition, many audio-to-text transcription services are now available via the Internet, providing a convenient and inexpensive way for users to obtain text-based transcriptions of audio data. One of the issues with conventional audio recording devices and audio-to-text transcription is that it is difficult to acquire and add visual content, such as images, video, etc., to the resulting text. Audio data and visual content are acquired separately, sometimes using separate devices, and the resulting data remains separate, leading to an unfavorable user experience.

SUMMARY

An apparatus is provided for processing audio data to generate transcription data that is supplemented with visual content items. The apparatus comprises one or more processors and one or more memories that store instructions which, when processed by the one or more processors, cause the apparatus to perform various functionality, including retrieving audio data that represents a plurality of spoken words, causing the audio data to be processed to generate transcription data that provides a textual representation of the audio data, identifying one or more specified keywords contained in the transcription data, wherein each specified keyword from the one or more specified keywords indicates a location in the transcription data where a visual content item is to be added to the transcription data, and generating revised transcription data by adding a visual content item or a reference to the visual content item. The approach may also be implemented by one or more computer-implemented methods or one or more non-transitory computer-readable media that store instructions which, when processed by one or more processors, cause the functionality to be performed.

DETAILED DESCRIPTION

II. System Architecture

III. Processing Audio Data and Adding Visual Content ItemsA. Adding Visual Content Items to Transcription DataB. Specialized KeywordsC. Adding Visual Content Items Based Upon Time

IV. Implementation Examples

Several approaches are provided for processing audio data to generate transcription data that is supplemented with visual content items. The visual content items may be any type of data that may vary depending upon a particular implementation. Examples of visual content items include, without limitation, images, videos, symbols, etc. Embodiments include adding visual content items to transcription data based upon user input, specialized keywords contained in the transcription data and various correspondences with the audio data, including time-based correspondence and correspondences based upon a common user, storage location or logical entity.

II. System Architecture

FIG. 1is a block diagram that depicts an arrangement100for processing audio data. Arrangement100includes an audio recording device110, client devices112,114and an audio data processing system120. All of the elements depicted inFIG. 1may be communicatively coupled with each other via one or more direct and/or indirect communications links. The communications links may be implemented by any number and type of wired or wireless networks, such as local area networks (LANs), wide area networks (WANs), the Internet, etc. Arrangement100may include fewer elements or more elements than depicted inFIG. 1, depending upon a particular implementation.

Audio recording device110is a device that is capable of recording human speech and generating audio data that represents the human speech. The audio data may be in any audio format that may vary depending upon a particular implementation, and embodiments are not limited to any particular audio format. Examples of audio formats include, without limitation, Windows Media Audio (.wma), WAV (.wav), Real Audio (.ra, .ram, .rm), Musical Instrument Digital Interface (.mid), and Ogg (.ogg). Audio recording device110may be implemented, for example, as a stand-alone recorder, or as a feature of a multi-feature device, such as a camera, mobile phone, tablet computing device, personal digital assistant (PDA), etc. Alternatively, audio recording device110may be integrated into a client device, such as client device112. Embodiments are also applicable to audio recording device110being capable of recording audio as part of video content. Example formats include, without limitation, .mp4 and .mov.

Client devices112,114may be implemented by any type of client device and embodiments are not limited to particular types of client devices. Examples of client devices112,114include, without limitation, a mobile communications device such as a smart phone, a tablet computing device, a personal digital assistant, a laptop computer, a desktop computer, a workstation, etc.

Audio data processing system120is a system that provides processing of audio data as described herein. This includes causing audio data to be processed to generate transcription data that provides a textual representation of the audio data, and generating revised transcription data by adding visual content items to the transcription data, as described in more detail hereinafter. Audio data processing system120may be implemented by computer hardware, computer software, or any combination of computer hardware and computer software. In the example depicted inFIG. 1, audio data processing system120includes a transcription service122that is configured to process audio data128and generate transcription data130that provides a textual representation of spoken words in the audio data. Transcription data130may be in any format that may vary depending upon a particular implementation and embodiments are not limited to transcription data being in any particular format. Example formats include, without limitation, plain text, rich text, etc. Audio data processing system120also includes a transcription data processing service124that is configured to process transcription data130and generate revised transcription data134by adding display content items, represented by visual content item data132, to transcription data130.

III. Processing Audio Data and Adding Visual Content Items

A variety of approaches may be implemented to supplement transcription data with display content item data. Several example approaches are described hereinafter.

A. Adding Visual Content Items to Transcription Data

According to one embodiment, visual content items are added to transcription data based upon a correspondence between audio data and visual content items. The correspondence may exist due to a wide variety of factors that may vary depending upon a particular implementation. Example factors that may establish a correspondence between audio data and visual content items include, without limitation, one or more of a user, a storage location, or a logical entity in common with both audio data and visual content items. Both audio data and visual content items may be generated by the same user as indicated, for example, by metadata for the audio data and the visual content items. For example, audio data may represent a recording of notes dictated by a physician for a particular patient. Both the audio data and visual content items, such as images, X-rays, etc., may correspond to the particular patient by a patient identifier for the particular patient. As another example, audio data and visual content items may be stored in the same location or correspond to the same project, group or organization. In the prior example, both the audio data and the visual content items may be stored in a folder or directory for the particular patient. Visual content items may be added to transcription data at specified locations, such as the beginning, middle, or end of transcription data. The specified locations may be specified by an administrative user or a user who acquires the audio data.

Consider the following example with reference toFIG. 1, the flow diagram200ofFIG. 2andFIGS. 3A-3C. In step202, audio data is generated. For example, a physician may use audio recording device110to dictate notes pertaining to a particular patient and audio recording device110generates audio data300(FIG. 3A). In this example, the physician dictates the phrase “Patient suffers from condition X as indicated in image. Recommend follow-up in two weeks.” It is presumed that audio data300generated by audio recording device110is transmitted to audio data processing system120and stored in audio data202, for example, as a particular audio file. This may be accomplished, for example, by the physician selecting an option on audio recording device110to upload audio data300to audio data processing system120. This may be done directly from audio recording device110to audio data processing system120, or via client device112.

In step204, audio data300is processed and transcription data310(FIG. 3B) is generated from audio data300. For example, transcription service122may process audio data300and generate transcription data310that is a text-based representation of audio data300. In the present example, transcription data310provides a textual representation of the notes dictated by the physician. Transcription data310may be stored as part of transcription data130, for example as a transcription data file, and may be in any of a wide variety of formats that may vary depending upon a particular implementation. For example, transcription data310may be in a text-based format or a rich text-based format.

In step206, revised transcription data320(FIG. 3C) is generated by adding one or more visual content items330to transcription data310. For example, transcription data processing service124may add one or more visual content items330, represented by visual content item data132, to transcription data310to generate revised transcription data320that is stored in revised transcription data134. Visual content items330may include, for example, medical records, images, video content, etc. In the current example, revised transcription data320may include both the notes dictated by the physician, as well as one or more images, such as photos, X-rays, MRIs, other scans, etc., for the particular patient. Visual content items330may be added to any location in the revised transcription data134, for example, at the beginning or end, or at one or more other specified locations. Revised transcription data320generated in step206may be in a format that is the same or different than transcription data310. The way in which visual content items are added to revised transcription data320may vary depending upon a particular implementation and the format of revised transcription data320. For example, some data formats may natively support both text and image data and for these formats, visual content items in the form of image data may be added to the revised transcription data320without requiring any special formatting. Other data formats may require special formatting or data conversions to be applied, either to the text, the visual content items, or both the text and the visual content items. This may also be applicable to other types of visual content items, such as video-based visual content items.

As an alternative to adding visual content items to the revised transcription data220, a reference or link to visual content items may be used instead. In the prior example, instead of the X-ray image, a reference to the X-ray image may be added to the revised transcription data220. The reference may be, for example, an address such as a URI.

According to one embodiment, the visual content items to be added to transcription data may be specified by a user. For example, transcription data processing service124may provide a graphical user interface that allows a user to select visual content items to be added to transcription data. The graphical user interface may also display transcription data and provide a visual indication of one or more locations where visual content items are to be added to the transcription data, and also provide controls that allow a user to select visual content items that are to be added at the one or more locations. The controls may allow a user to navigate to a location of stored visual content items and select particular visual content items to be included in the revised transcription data220. As a further enhancement, the graphical user interface controls may direct a user to visual content items that correspond to the audio data200. In the prior example, the user may be directed to a location that stores visual content for the particular patient.

B. Specialized Keywords

According to one embodiment, specialized keywords, also referred to herein as “specified keywords,” are used to specify locations where visual content items are inserted into transcription data. The specialized keywords may be any spoken word, or combination of words, that are capable of being recognized in transcription data. The specialized keywords may be words that have meaning in a particular language, or they may be words that do not have any meaning in a particular language.

In the prior example, the physician may speak the specialized keyword “VisiLink” while dictating the notes for the particular patient. The transcription data includes the keyword “VisiLink” at the location in the text where the physician spoke the keyword “VisiLink.” The transcription data processing service124inserts the visual content items in the revised transcription data134at the location of the keyword “VisiLink.” The visual content data may be inserted into the revised transcription data134before or after the specialized keyword “VisiLink,” or may replace the specialized keyword “VisiLink” with the visual content data by inserting the visual content data at the location of the specialized keyword “VisiLink” and remove the specialized keyword “VisiLink.” This use of specialized keywords allows an end user to specify where visual content items are added to transcription data.

According to one embodiment, specialized keywords are accompanied by visual content identification data that identifies visual content items to be added to transcription data. The visual content identification data may identify visual content items directly, for example, by name. Alternatively, the visual content identification data may indirectly identify visual content items, for example, by a reference or a location. Visual content identification data may take many forms and may vary depending upon a particular implementation. Example implementations of visual content identification data include, without limitation, names, filenames, identifiers, signatures, random numbers, guaranteed unique numbers, alphanumeric codes, etc.

Visual content identification data may be generated using a wide variety of techniques that may vary depending upon a particular implementation. According to one embodiment, a user of a device that acquires or creates visual content items, such as a camera, tablet computing device, smart phone, personal digital assistant (PDA), etc., may be configured to allow a user to manually specify visual content identification data for a visual content item. Referring to the prior example of the specialized keyword “VisiLink,” the device may include an “Add VisiLink” physical or graphical user interface control which, when selected, allows a user to manually enter visual content identification data for a visual content item that is about to be acquired, or that has previously been acquired. The user may determine the visual content identification data, or the visual content identification data may be determined by other personnel, such as assistants, etc. For example, in the medical context, visual content identification data may be determined by other physicians, clinicians, etc.

According to another embodiment, a device that acquires or creates visual content items may be configured to generate visual content identification data. This may be done automatically for every visual content item acquired, or may be manually requested by a user. For example, the device may include a “Generate VisiLink” control or button which, when selected, causes the generation of visual content identification data. The device itself may generate the visual content identifier or alternatively, the device may retrieve the visual content identifier from a remote location, for example, a server.

According to one embodiment, visual content identification data may be included in visual content items. For example, visual content identification data, e.g., in the form of a unique identifier, may be included directly in a visual content item, such as an image. The visual content identification data included in the visual content item may be in human-readable form. Alternatively, the visual content identification data may be in machine-readable form, such as encoded in a bar code, QR code, one or more other symbols, etc.

Visual content identification data may be unique to each visual content item, for example, by using long numbers, alphanumeric codes, signatures, etc. Alternatively, visual content identification data may only need to be unique with respect to other visual content items associated with a particular logical entity, such as a patient. This allows shorter visual content identifiers to be used.

The visual content identifier may be provided to audio data processing system120separate from the corresponding visual content item, or included with the corresponding visual content item, for example, in metadata for the corresponding visual content item.

Referring again to the prior example, suppose that the physician wants to add an X-ray for the particular patient into the transcription data. While dictating his notes, the physician speaks the specialized keyword “VisiLink,” followed by “123456” at the location where the physician would like the X-ray to be added. The “123456” is one non-limiting example of visual content identification data that identifies the visual content item(s) to be added to the transcription data. In the present example, the visual content identification data “123456” identifies the X-ray for the particular patient that the physician would like to add to the transcript. When the revised transcription data134is generated, the X-ray is added at the location of the specialized keyword “VisiLink.” For example, transcription data processing service124may locate, for example in visual content item data132, the visual content item that corresponds to the visual content identification data “123456,” and add the visual content item to the revised transcription data. Alternatively, transcription data processing service124may locate the visual content item in a location external to audio data processing system120. This approach allows an individual to verbally specify both the visual content items that are to be included in a transcription and the locations of those visual content items. Any number of visual content items may be specified with a single set of one or more specialized keywords. In the prior example, visual content identification data for a single visual content item follows the “VisiLink” specialized keyword for purposes of explanation, but visual content identification data for multiple visual content items may follow a specialized keywords. This may be useful in situations where a user wishes to have multiple visual content items inserted into the transcription data at a particular location.

FIGS. 4A-4Cdepict an approach for supplementing transcription data with visual content items, according to an embodiment. InFIG. 4A, a user, which in this example is a physician, uses audio recording device110to dictate notes pertaining to a particular patient and wants to add visual content items to the transcription data. In this example, the physician dictates the phrase “Patient suffers from condition X as indicated in image VisiLink 123456 and X-rays VisiLink 123457 123458 Recommend follow-up in two weeks.” The audio recording device110used by the physician generates audio data400that is transmitted to audio data processing system120and stored in audio data128. Audio data400covers a time period T1-T10and the specialized keyword “VisiLink” was spoken by the physician at time T5, followed by the visual content identification “123456” and at time T6, followed by the visual content identifications “123457” and “123458.”

FIG. 4Bdepicts transcription data410that is generated by transcription service122after processing audio data400to generate transcription data410that provides a textual representation of audio data400. The transcription data410includes the specialized keyword “VisiLink” and the visual content item identification data “123456,” “123457” and “123458.”

FIG. 4Cdepicts revised transcription data420generated by transcription data processing service124in which the specialized keywords and visual content item identification data have been replaced with the corresponding visual content items. More specifically, the specialized keyword “VisiLink” and visual content item identification data “123456” have been replaced with an image, and the specialized keyword “VisiLink” and visual content item identification data “123457” and “123458” have been replaced with two X-rays.

C. Adding Visual Content Items Based Upon Time

According to one embodiment, time-based correspondence is used to add visual content items to transcription data. This may include adding to transcription data, a visual content item that has a time that corresponds to a time of the audio data that was transcribed. For example, the visual content item may have an acquisition or creation time that is within a time range covered by the audio data that was transcribed. As another example, a visual content item may have an acquisition or creation time that is within a specified amount of time of a time range covered by audio data that was transcribed. The specified amount of time may be selected so that a reasonable correspondence is established between visual content items and audio data.

FIGS. 5A-5Cdepict an approach for supplementing transcription data with visual content items based upon time, according to an embodiment. InFIG. 5A, a user, which in this example is a physician, uses audio recording device110to dictate notes pertaining to a particular patient and wants to add visual content items to the transcription data. In this example, the physician dictates the phrase “Follow-up examination indicates significant improvement in condition X.” The audio recording device110used by the physician generates audio data500that is transmitted to audio data processing system120and stored in audio data128. Audio data500covers a time period T10-T15. As used herein, the notation “T” may represent any unit of time, for example, seconds, minutes, hours, etc.

FIG. 5Bdepicts transcription data510that is generated by transcription service122after processing audio data500to provide a textual representation of audio data500. The transcription data510provides the textual representation for the sentence “Follow-up examination indicates significant improvement in condition X.”

FIG. 5Cdepicts revised transcription data520generated by transcription data processing service124in which visual content items have been added to transcription data. The visual content items may be added at any location in revised transcription data520. InFIG. 5C, arrows indicate example locations where visual content items may be added. Visual content item data530includes data for five visual content items, where each row corresponds to a particular visual content item. Each visual content item has an item number in the form of an integer, a visual content item identifier (ID) in the form of an alphanumeric string, an acquisition or creation time, and a type, such as image, X-ray, video, etc.

According to one embodiment, visual content items having a creation time that is within the time range covered by audio data500are included in revised transcription data520. In the present example, visual content item2has a creation time of T13that is within the T10-T15time range covered by audio data500and is therefore included in revised transcription data520. According to another embodiment, visual content items that have an acquisition or creation time that is within a specified amount of the time range covered by audio data500are included in revised transcription data520. For example, suppose that the specified amount, which may also be considered to be a threshold, is set at 5. In this example, the third and fourth visual content items represented in visual content item data530would also be included in revised transcription data500, because the acquisition or creation times of T20and T9, respectively, are within 5 units of the T10-T15time range covered by audio data500. The specified amount or threshold may be selected to provide useful results in various contexts. A specified time or threshold that is too short may miss some visual content items, while a specified time or threshold that is too long may cause visual content items to be incorrectly associated with audio data. For example, suppose that in a medical context, the typical dictation made by a physician is tens of seconds to a few minutes in duration. In this situation, the specified time or threshold may be set to five or ten minutes so that visual content items will be associated with the correct corresponding audio data for a patient, while providing adequate discrimination between patients. In this example, using a specified time or threshold that is too short may cause some visual content items to be missed, i.e., not associated with a patient, while using a specified time or threshold that is too long may cause some visual content items to be associated with audio data for another patient.

Visual content items may be added at a single location in revised transcription data520, or at multiple locations within revised transcription data520. For example, visual content items may be grouped together at the beginning or end of revised transcription data520. According to one embodiment, visual content items are added to locations in revised transcription data520that correspond to the acquisition or creation time of the visual content items. For example, visual content item number two would be added to revised transcription data520at a location that corresponds to time T13. This may be accomplished, for example, by determining times at which each of the words in revised transcription data520occurred in audio data500, and then inserting the visual content item at the appropriate location.

IV. Implementation Examples

Although the flow diagrams of the present application depict a particular set of steps in a particular order, other implementations may use fewer or more steps, in the same or different order, than those depicted in the figures. The embodiments described herein provide user friendly and flexible approaches for adding visual content items to transcription data. The approaches allow visual content items to be automatically added to transcription data based upon user input, specialized keywords contained in the transcription data and various correspondences with the audio data, including time-based correspondence and correspondences based upon a common user, storage location or logical entity. The approaches also improve the performance of computers on which the approaches are implemented at least by reducing the amount of computational resources required to generate revised transcription data that includes visual content items.

FIG. 6is a block diagram that depicts an example computer system600upon which embodiments may be implemented. Computer system600includes a bus602or other communication mechanism for communicating information, and a processor604coupled with bus602for processing information. Computer system600also includes a main memory606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus602for storing information and instructions to be executed by processor604. Main memory606also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor604. Computer system600further includes a read only memory (ROM)608or other static storage device coupled to bus602for storing static information and instructions for processor604. A storage device610, such as a magnetic disk or optical disk, is provided and coupled to bus602for storing information and instructions.

Computer system600may be coupled via bus602to a display612, such as a cathode ray tube (CRT), for displaying information to a computer user. Although bus602is illustrated as a single bus, bus602may comprise one or more buses. For example, bus602may include without limitation a control bus by which processor604controls other devices within computer system600, an address bus by which processor604specifies memory locations of instructions for execution, or any other type of bus for transferring data or signals between components of computer system600.

Computer system600may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic or computer software which, in combination with the computer system, causes or programs computer system600to be a special-purpose machine. According to one embodiment, those techniques are performed by computer system600in response to processor604processing instructions stored in main memory606. Such instructions may be read into main memory606from another non-transitory computer-readable medium, such as storage device610. Processing of the instructions contained in main memory606by processor604causes performance of the functionality described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the embodiments. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

The term “non-transitory computer-readable medium” as used herein refers to any non-transitory medium that participates in providing data that causes a computer to operate in a specific manner. In an embodiment implemented using computer system600, various computer-readable media are involved, for example, in providing instructions to processor604for execution. Such media may take many forms, including but not limited to, non-volatile and volatile non-transitory media. Non-volatile non-transitory media includes, for example, optical or magnetic disks, such as storage device610. Volatile non-transitory media includes dynamic memory, such as main memory606. Common forms of non-transitory computer-readable media include, without limitation, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip, memory cartridge or memory stick, or any other medium from which a computer can read.

Various forms of non-transitory computer-readable media may be involved in storing instructions for processing by processor604. For example, the instructions may initially be stored on a storage medium of a remote computer and transmitted to computer system600via one or more communications links. Bus602carries the data to main memory606, from which processor604retrieves and processes the instructions. The instructions received by main memory606may optionally be stored on storage device610either before or after processing by processor604.

Computer system600also includes a communication interface618coupled to bus602. Communication interface618provides a communications coupling to a network link620that is connected to a local network622. For example, communication interface618may be a modem to provide a data communication connection to a telephone line. As another example, communication interface618may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface618sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Computer system600can send messages and receive data, including program code, through the network(s), network link620and communication interface618. In the Internet example, a server630might transmit a requested code for an application program through Internet628, ISP626, local network622and communication interface618. The received code may be processed by processor604as it is received, and/or stored in storage device610, or other non-volatile storage for later execution.

In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is, and is intended by the applicants to be, the invention is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.