METHOD AND SYSTEM FOR VOICE TO TEXT REPORTING FOR MEDICAL IMAGE SOFTWARE

A system and method for voice to text reporting for medical image software. The system and method may optionally include a separate voice to text engine, for converting the voice report to text, and also some type of medical image software, for providing medical image processing capabilities. According to at least some embodiments, both capabilities are provided remotely to the user's computer, and may optionally be provided through a “zero footprint” on the user's computer.

DESCRIPTION OF EMBODIMENT OF THE INVENTION

Although the present invention is described with regard to a “computer” on a “computer network”, it should be noted that optionally any device featuring a data processor and the ability to execute one or more instructions may be described as a computer, including but not limited to any type of personal computer (PC), a server, a cellular telephone, an IP telephone, a smart phone, a tablet, a PDA (personal digital assistant), or a pager. Any two or more of such devices in communication with each other may optionally comprise a “computer network”.

Although the present description centers around medical image data, it is understood that the present invention may optionally be applied to any suitable three dimensional image data, including but not limited to computer games, graphics, artificial vision, computer animation, biological modeling (including without limitation tumor modeling) and the like.

At least some embodiments of the present invention are now described with regard to the following illustrations and accompanying description, which are not intended to be limiting in any way.

FIG. 1Ashows an exemplary, illustrative system according to at least some embodiments of the present invention for voice to text reporting for medical image software. As shown, a system100features a plurality of user computers102(shown as user computers1-3102for the sake of illustration only and without any intention of being limiting), two of which are shown as operating a web browser104, again for the sake of illustration only and without any intention of being limiting. Web browser104is a non-limiting example of a software program, capable of communicating according to HTTP and rendering HTML (HyperText Markup Language); any suitable software program or “app” could be used in its place, for example if user computer102were to be implemented as a “smartphone” or cellular telephone with computational abilities.

User computer1102is in communication with a remote server108through a computer network106. Computer network106may optionally be any type of computer network, such as the Internet for example. For the sake of security, computer network106preferably features at least a security overlay, such as a form of HTTPS (secure HTTP) communication protocol, or any type of security overlay to the communication protocol, such as 256-bit SSL3 AES and security certificates for example, and may also optionally feature a VPN (virtual private network) in which a secure “tunnel” is effectively opened between user computer102and remote server108.

It should be noted that remote server108may optionally comprise a plurality of processors and/or a plurality of computers and/or a plurality of virtual machines, as is known in the art.

Remote server108optionally and preferably operates an HTML server130as well as a medical image processing software, shown herein as PACS module110, although any suitable medical image processing software may optionally be provided, for example which operates according to DICOM (Digital Imaging and Communications in Medicine). PACS module110may optionally comprise any type of medical image processing software or a combination of such softwares. PACS module110is preferably in communication with a remote server132which may be a PACS server or a DICOM archive. Remote server132stores the medical images in storage136and also comprises a database112for holding medical image data.

Database112is shown herein as being incorporated into remote server132but may optionally be incorporated into remote server108or may be separate from these servers (not shown). Remote server108communicates with remote server132through a computer network140, which may optionally be implemented as described with regard to computer network106, optionally and preferably including the same or similar security features.

PACS module110processes medical image data, for example allowing images to be segmented or otherwise analyzed; supporting “zoom in-zoom out” for different magnifications or close-up views of the images; cropping, highlighting and so forth of the images. HTTP server130operating on server108preferably renders the Web interface of the PACS module110in HTML so that Web browser104can display a PACS interface through which the user can perform such actions and view results using user computer102. Optionally the actions are performed locally at user computer102but are preferably performed at remote server108.

Optionally and more preferably, PACS module110provides complete support for medical image processing, such that the medical image processing software has “zero footprint” on user computer102or on web browser104, such that optionally and more preferably not even a “plug-in” or other addition to web browser104is required. In other words, web browser104does not feature a process associated plugin, meaning a plugin that is associated with or operated by the medical image processing software. Such complete support for remote medical image viewing and analysis is known in the art, and is in fact provided by the Vue Motion product currently being offered as part of Carestream Health offerings. All of these examples relate to examples of “thin clients”, with low or “zero” footprints on user computer102, preferably provided through a web browser but optionally provided through other software.

However, currently medical image processing software, while providing support for such remote medical image viewing and analysis, does not provide support for voice to text report generation, nor does it provide support for combining such generated reports with the medical images that were viewed by the doctor while generating the report. System100overcomes these drawbacks of the background art by also providing a remote server114, which operates a voice to text engine116. Voice to text engine116may optionally be any such engine which is known in the art, including but not limited to such engines that are available from Nuance (for example and without limitation, the360SpeechAnywhere platform). Voice to text engine116may also optionally feature a dictionary118as shown, which may optionally and preferably comprise specialized medical terms, of the type that are likely to be of interest or needed for dictating a medical image diagnostic report. Remote server114communicates with user computer through a computer network130, which again may optionally be implemented as described with regard to computer network106, optionally and preferably including the same or similar security features.

The user preferably interacts with voice to text engine116as follows. The user, such as a doctor for example, reviews medical images through web browser1104, being operated by user computer1102, in communication with remote server108. As the user reviews these medical images, the user dictates a report through a microphone or other voice collecting device on user computer1102(not shown). The voice data is then transmitted from user computer1102to remote server114, for processing by voice to text engine116. Voice to text engine116then transmits back a text report to the user. The converted text is preferably transmitted back for viewing as the user dictates or is at least transmitted back intermittently, such that the user views dictated text in near real time. Alternatively, the text is transmitted back when the user completes their dictation. Optionally and preferably, voice to text engine116transmits a list of words matching the dictation, while the actual generation of the report (and hence preferably also editing of the report) is performed through web browser104.

In addition to being viewed, the text may be optionally edited through web browser1104for example (acting as a zero footprint PACS user interface), or alternatively through any type of word processing software (not shown); for example, voice to text engine116may optionally use a secure channel to transmit back the written report. The user may then optionally change the report manually, by typing on the computer keyboard of user computer1102(not shown) for example, before the report is transmitted to database112.

As an additional security measure, optionally neither the voice data nor the resultant text data is stored on remote server114; in other words, optionally a session is set up to connect user computer1102and remote server114as necessary for creating the text report, with data being maintained only in a temporary memory on remote server114and not in a permanent database. Once the session has been closed, for example once the user is finished with at least the dictation part of the report generation process, then any temporarily stored data on remote server114is preferably flushed and is not stored permanently. However, dictionary118may optionally be an exception to this rule, as dictionary118may optionally learn from a particular user or from a plurality of users, and incorporate corrections or changes made by the user on a permanent basis.

With regard to the communication between user computer1102and remote server114, optionally the “zero footprint” standard is maintained, such that all support for such communication effectively occurs through web browser1104. Otherwise, some type of user interface software would need to be present on user computer1102, for supporting communication with voice to text engine116(not shown). The user interface enabling control of the dictation and voice to text process on Web browser1104, is provided by remote server108.

The operation of system100is described in greater detail with regard toFIGS. 2A and 2B, but briefly system100may optionally operate as follows. The user views medical images through web browser1104, supported by PACS module110. As the user views these images, the user verbally dictates a report, which may optionally be transmitted simultaneously or only after dictation is completed to remote server114. As the user dictates the report, or optionally after dictation is complete, the user may optionally select one or more medical images for being combined with the report through web browser1104. For example, the user may optionally request that a particular image be included through “bookmarking” the image through an interaction with web browser1104; the user may also optionally request that the entire image be included or only a link to the image (for example, to reduce the size of the final report). Optionally, any image that the user views while recording the dictated report may be automatically included; alternatively or additionally, some combination of these features may optionally be used to somehow connect, combine, bundle or link one or more images with the report. It is also possible to include all images in the final report.

As described above, the Voice to text engine116then transmits back a text report to the user, for being viewed and optionally edited through web browser1104for example (acting as a zero footprint PACS user interface), or alternatively through any type of word processing software (not shown). The user may then optionally change the report manually, by typing on the computer keyboard of user computer1102(not shown) for example.

Once the user is satisfied that the text is correct and the appropriate images have been included and the report is therefore complete, the user optionally and preferably “signs off” or otherwise indicates the report's completed state through web browser1104. This information is then transmitted to remote server108, which optionally and preferably stores a copy of the report in database112and/or in a separate DICOM archive such as in storage136as previously described, more preferably along with an indication of the report's connection to various images. Optionally the report may be stored in a Radiology Information System or in a Hospital Information System.

Optionally, an additional user may request to view the report through user computer2102, operating web browser2104. Alternatively, in fact the same user may request to view the report but through a different computer. User computer2102is preferably in communication with remote server108through a computer network120, which may optionally be implemented as described previously for computer network106. Web browser2104enables the user to retrieve the report from remote server108(for example from database112) and to make any edits or changes, or comments; the user may then optionally sign off on the report or may alternatively pass the report to another user for signing off. Optionally and preferably, all such communication regarding the report passes through remote server108for security purposes; furthermore, by passing through remote server108, optionally and preferably the images themselves do not need to be sent as part of the report (although they can be).

Although the previous description centered around user computers102which supported “zero footprint” interactions with remote server108through web browsers104, in fact optionally a user computer3102may feature a PACS viewer124as shown. PACS viewer124features some or all of the functionality of PACS module110for image processing, analysis and manipulation. The user operating user computer3102may therefore optionally change one or more of the images through local processing by PACS viewer124on user computer3102as shown. PACS viewer124may also optionally feature its own image database (not shown). User computer3102is preferably in communication with remote server132through a computer network122, which may optionally be implemented as described previously for computer network106.

Each of user computer2102and user computer3102may optionally be in contact (not shown) with remote server114in order to be able to interact directly with voice to text engine116.

It should be noted that although computer networks106,120,122,130and140are described as being separate networks, in fact any plurality of such networks, or even all such networks, may optionally be comprised in a single network.

FIG. 1Bshows another exemplary, illustrative system according to at least some embodiments of the present invention for voice to text reporting for medical image software. The operation of this embodiment of system100is similar to that ofFIG. 1A, except that access to voice to text engine116is provided through remote server108, whether operated (not shown) by remote server108or operated by remote server114which communicates with all user computers102through remote server108as shown. In this embodiment, remote server114features an engine interface150, which supports interactions between remote server108and voice to text engine116. The “zero footprint” can still be maintained at user computers102, as instead the voice to text support functionality is shifted to remote server108and/or remote server114.

FIGS. 2A and 2Bshow exemplary, illustrative processes according to at least some embodiments of the present invention for voice to text reporting for medical image software.

FIG. 2A, as shown, relates to an exemplary process for an emergency situation, for supporting the generation of a written medical image diagnostic report. A process200starts with a patient being scanned on an emergency basis in stage202; the medical images are then uploaded to some type of PACS-enabled server in stage204. In stage206, the radiologist (or other doctor) that is on-call is asked to provide a diagnostic analysis of the medical image data. It should be noted that the doctor may optionally be located remotely from the PACS-enabled server and may not in fact even have access to a computer with a local PACS module, but may instead perform the below stages through a remote computer, tablet or smartphone, for example optionally through the above described zero footprint implementation.

In stage208, the doctor reviews the medical images and dictates the report (for example by using the system as described above with reference toFIGS. 1A and 1B).

After dictation is complete, the doctor may optionally select one or more medical images for being combined with the report. For example, the doctor may optionally request that a particular image be included through “bookmarking” the image; the doctor may also optionally request that the entire image be included or only a link to the image (for example, to reduce the size of the final report). Optionally, any image that the doctor views while recording the dictated report may be automatically included; alternatively or additionally, some combination of these features may optionally be used to somehow connect, combine, bundle or link one or more images with the report. It is also possible to include all images in the final report.

In stage209the dictated report is converted to text using the voice to text process including review, correction, and editing by the doctor as described with reference toFIGS. 1A and 1Babove. The doctor may then optionally either ‘save as a draft’ or ‘sign’ the report (usually as preliminary). In stage210, the report which includes both images or links to images and the approved text is then stored through the previously described remote server with PACS module, and is available for another doctor to continue the reporting process, optionally also using the speech or text process, until a final report is available. As shown in this non-limiting example, the process continues with a senior radiologist's review in stage212, leading to finalization of the report in stage214.

Among the advantages of this process (but without wishing to enumerate a closed list) are that none of the doctors involved need to be at the same physical location, nor do they need to be in direct communication by telephone, email and so forth. Instead the process200permits different doctors to comment and report at different times, and also permits a senior doctor (such as a senior radiologist for example) to control when the report is finalized, and hence to control process200. The voice to text mechanism described above is an integral part of this process and offers the desired advantages as outlined in the summary of the invention such as speeding up the report generation process while reducing the potential for errors in the dictation process. Additionally, the functions described above are part of an integrated system.

Other safeguards and requirements may also optionally be built into process200, which are not necessarily automatically available today, such as the requirement for at least one doctor to review the report before it can be signed as final. Furthermore, these advantages are available in an emergency situation, which by its very nature is not planned and so which can strain manually implemented processes.

FIG. 2Bshows an exemplary process for supporting the generation of a written medical image diagnostic report by a resident, which is then finalized after review by a more senior physician. A process250starts with a patient being scanned on any basis (and not necessarily an emergency basis) in stage252; the medical images are then uploaded to some type of PACS-enabled server in stage254. In stage256, the resident reviews the medical images and generates a preliminary report through dictation (for example as described above). At stage257, the dictated report is converted into a text based report using the systems as described above. It should be noted that the doctor may optionally be located remotely from the PACS-enabled server and may not in fact even have access to a computer with a local PACS module, but may instead perform the below stages through a remote computer, tablet or smartphone, for example optionally through the above described zero footprint implementation.

In stage258, the preliminary report is stored in text form along with associated images through the previously described remote server with PACS module. In stage260, the attending physician is able to review the report, with or without access to a local PACS module as previously described. In stage262, the attending physician determines whether the report is accurate. If the attending physician decides that the report is generally accurate, then in stage264, the attending physician makes any comments or changes, optionally using the speech to text capabilities, and signs the report. In stage266, the final report is made available, again optionally through the above described remote server and PACS enabled system.

However, if the attending physician feels that any/significant changes need to be made to the report, then from stage262the process instead continues to stage268, in which the attending physician requests various changes to the report from the resident, optionally using the speech to text capabilities. In stage270, the preliminary report is returned for the resident to continue to work on it, and the process continues at stage258. This cycle may optionally continue until the final report is made available in stage266.

Again, process250has advantages over fully manual processes, in that again (without wishing to be limited by a closed list), the resident and the attending physician do not need to be at the same physical location, nor do they need to be in direct communication by telephone, email and so forth. The process250permits different doctors to comment and report at different times, and also permits a senior doctor (such as a senior radiologist for example) to control when the report is finalized, and hence to control process250. The voice to text system here again offers the advantages outlined above.

Other safeguards and requirements may also optionally be built into process250, which are not necessarily automatically available today, such as the requirement for at least one doctor to review the report before it can be signed as final. Furthermore, doctors or other users may be present at widely separated locations and indeed may optionally interact through process250from any type of location and also through any type of suitable electronic device, optionally including but not limited to mobile or portable electronic devices.

FIG. 3shows an exemplary, illustrative process300for the operation of the systems ofFIGS. 1A and 1Baccording to at least some embodiments of the present invention.FIG. 3illustrates optional sources and inputs that comprise a report such as those described with reference to the embodiments above.

As shown, one or more different sources may be used to provide information for creating a report380, which at the end of the process becomes a signed report (at390) that is stored in the PACS. The sources may include text which is a translation of the dictation of the user, for example as described above and shown in302-306, text that has been added manually by the user or edited following the voice to text process, as shown at308, and one or more medical data elements which are received and/or selected by the user. For example: the user may add clinical reports (at320), such as structured reports generated by modalities (imaging equipment) such as DICOM SR (structured reporting), vessel analysis and calcium scoring reports; select key images from the medical imaging studies (at322); and/or add measurements and image annotations which are related to her diagnosis, as shown at324.

Optionally, a medical imaging study or segments of the study in the form of one or more images therefrom (at322) may be added to the report as decided by the user. Optionally, the segments, which are added to the report, define anatomic sites, each referred to in the dictation or text accompanying these segments. In such a manner, the report may provide a visual reference to the diagnosis of the user. Optionally, the above described PACS viewer and/or web browser provided image viewer allows the user to mark anatomical sites on the segments of the medical imaging study (as at324) which are added to the report, optionally in the form of bookmarks that can then be inserted into the text such that a user viewing the text can select a bookmark and be shown the marked site on the image. In such a manner, the user may refer the reader to specific areas of interest by pointing out the marked sites.

Optionally, the above described voice to text process, as at304, may be used for identifying references to anatomical sites defined by the user. In such an embodiment, the user may optionally select segments of the imaging study at322according to the identified anatomical sites and add them to the report in association with a respective section in the diagnosis. Alternatively or additionally the user may mark segments of the imagining study as at324according to the identified anatomical sites and associate them to respective sections of the report. Optimally, the user may include a key-phrase in his/her voice dictation that will be interpreted by the voice to text process as an instruction to add a link to a defined bookmark in the converted text. The bookmark function is described above.

Optionally, the above described PACS module is connected to a computer aided diagnosis (CAD) system330. In such an embodiment, the CAD system330may receive and process one or more diagnosed medical imaging studies and output an automated analysis accordingly. Optionally, the automated analysis is added to the report, at system330, and/or used to automatically update of the report.

According to some embodiments of the present invention, the imaging study is presented to the user according to a protocol which has been selected according to the modality and/or the anatomical site which is related thereto. Optionally, the imaging study comprises a set of views, such as posterior, anterior, lateral, superior and/or interior views. In such an embodiment, the views may be presented sequentially. Each presented view allows the user to relate thereto and to determine when to present the following view. Optionally, the views are added in a sequential manner to the report, optionally each with an association to the related diagnosis which has been provided by the user. In such a manner, the report that is outputted in the end of the medical reporting session, for example as shown at signed report390, may be generated in a manner such that each diagnosis is presented with the view on which it is based. Optionally, the sequence is dynamically adjusted according to the behavior of the user.

As shown at380, the report is created based on the possible sources combined with the text diagnosis. Optionally, as shown at390, the report is signed, for instance with a digital signature. Optionally, the signed report is forwarded at forwarding process395, as previously described with reference toFIGS. 2A and 2B, for comments and/or approval and/or to a report database, such as database112ofFIGS. 1A and 1B.

The generated report, as produced by process300includes rich content such as text, measurements, image notations/markings and bookmarks to these, and images. Optionally, the reports further comprise rich data such as hyperlinks, tables, and graphs which are based on a combination of inputs from the user and/or the received medical imaging studies and/or medical records added at other sources process332.

FIG. 4shows an exemplary, illustrative method according to at least some embodiments of the present invention for documenting an informal workflow. By “informal workflow” it is meant a workflow that does not necessarily end in the production of a diagnostic or medical report, or where the information flow is not documented in any digital system. For instance, in ER scenarios a patient is scanned, the doctor contacts the radiologist by phone to review the images and provide an opinion. The radiologist review the images, provides the opinion but no record of the conversation or the radiologist's opinion is stored anywhere. As shown, in stage1, an opinion is requested of a physician regarding a medical image study or alternatively a portion of such a study, comprising one or more images. The request may optionally be sent through a computer network, for example by email, or alternatively may optionally be made verbally.

In stage2, the physician views one or more images, comprising part or all of an image study, according to the request (which may optionally direct the physician to the specific image(s) or study, or alternatively may optionally refer to the patient for example) through a viewing application as described above, whether a PACS viewer or a “thin client” viewer (for example provided through a web browser as described herein). The viewing application may optionally be provided through a computer or cellular telephone (such as a smartphone) or other electronic device as described above.

In stage3, as the physician views the one or more images, the physician dictates a verbal (i.e.—voice) report to the electronic device, which is preferably the same electronic device that is displaying the one or more images.

In stage4, the verbal (i.e.—voice) report is converted to text as previously described. In stage5, text is optionally added to, deleted from, or changed within the report through any suitable mechanism, including but not limited to additional verbal information that is converted to text, manually editing the reporting, manually or automatically adding, deleting, changing or editing text, and so forth.

In stage6, the verbal report is preferably stored in association with the one or more images, or image study, thereby enabling the opinion and thoughts of the physician to be captured and to be made part of the permanent record regarding the image(s) viewed.

FIGS. 5A and 5Bshow exemplary, illustrative screenshots according to at least some embodiments of the present invention. The screens show the medical image viewing and reporting application in a Web browser501. The right pane502comprises a Web enabled radiology reporting interface, with various elements required to implement the embodiments described above. These elements include a record button503for initiating the voice to text process; a sign button504allowing the practitioner to digitally sign the report; and a text editor510for adding text or reviewing and editing text that has been converted from voice. As shown, the radiologist would typically manipulate the controls of the radiology reporting functions on the right502while viewing a medical image508on the left.FIG. 5Bshows text editor510following conversion of a spoken diagnosis into text.

Although the present description centers around interactions with medical image data, it is understood that the system may be applied to any suitable three dimensional image data, including but not limited to computer games, graphics, artificial vision, computer animation, biological modeling (including without limitation tumor modeling) and the like.

Although the present invention is described with regard to a “computer” on a “computer network”, it should be noted that optionally any device featuring a data processor and the ability to execute one or more instructions may be described as a computer, including but not limited to any type of personal computer (PC), a server, a cellular telephone, an IP telephone, a smart phone, a PDA (personal digital assistant), or a pager. Any two or more of such devices in communication with each other may optionally comprise a “computer network”.