Patent ID: 12191027

DETAILED DESCRIPTION OF ILLUSTRATED EMBODIMENTS

Systems, methods, computer program products and interactive computer-generated user interfaces for snip-triggered generation of digital image reports during review or navigation of images presented through a display of an image review workstation. Embodiments provide for automated generation of an image report or portion thereof in response to a pre-determined user input during image review and during review of an image of interest while eliminating or reducing user input required to create a report while eliminating or reducing image review workflow interruptions. Embodiments execute in-line and in real time during image review to improve diagnostic image review workflows conducted by radiologists and troubleshooting image review workflows conducted by engineers.

Embodiments provide for automated digital image report generation during an image review workflow thus resulting in more efficient, convenient and comprehensive image review and report generation that also assists with reducing user errors while eliminating or reducing user input. Improvements provided by embodiments are further enhanced by snip-triggered capture of various types of pertinent data and generation of an image report including same such as selected metadata of an image file such as a Digital Imaging and Communications in Medicine (DICOM) file. In this manner, only certain metadata of interest to the user, e.g., a subset of the hundreds of different types of metadata in a DICOM file header, are automatically incorporated into a digital image report together with the selected portion of the currently displayed image and other associated images if so configured. Embodiments provide for image review workstations that are more efficient and user friendly and that generate more comprehensive digital image reports to facilitate image reviews and reduce user errors without disrupting the review workflow as a result of executing while the user is already interacting with the currently displayed image of interest. In this manner, the user is not required to stop and start review to execute another program or record notes regarding the currently displayed (or other) image.

Embodiments provide for transmission of the digital image report to another computer device. The other computer device can be either an image review workstation or any other device that allows viewing of the report generated by using the snip-triggered capture. In this way, a radiologist creating a radiology report can pass on the information gleamed from the review to another reviewer. Radiologists commonly spend a significant amount of time reviewing multiple images including different image views, imaging modalities and image manipulations as part of the diagnosis of cancer. Another reviewer subsequently picking up a patient case, for example for purposes of second review, biopsy, or additional imaging, would have to duplicate the same reviewing of different image views, imaging modalities and image manipulations in order to find the same region of interest identified by the first reviewer. Such work is duplicative and inefficient. By using the snip-triggered capture, any subsequent reviewer would have the information for the reasons for the first radiologist's diagnosis as well as the snip-triggered image of the region of interest in one report.

In addition, the comprehensive digital image reports can be used as part of patient reporting obligations. After the radiologist reaches the diagnosis, the diagnosis is communicated to the patient. The report communicated would be improved by including a specific image of interest as part of the report. The patient would be benefited from the information and could also use that information for subsequent diagnosis and treatment, partially by another location.

Exemplary medical imaging environments in the form of breast image analysis systems in which embodiments may be implemented are described with reference toFIGS.1-2, and embodiment configurations and operation relative to these exemplary systems and environments are described with reference toFIGS.3-10.

FIG.1depicts one example of a medical imaging environment or breast image analysis system100(generally, breast image analysis system) that may be adapted, configured or programmed to execute embodiments of snip-triggered digital image report generation.FIG.1depicts breast image analysis system100including an image acquisition device110and an acquisition workstation120in communication with each other through a communication link or network130. Image acquisition device110may utilize various imaging modalities. One example of image acquisition device110is a tomosynthesis acquisition system.

One exemplary image acquisition device110as depicted inFIG.1includes a rotatable tube arm assembly112carrying an x-ray tube assembly114. Other imaging devices may be utilized, but for ease of explanation,FIG.1is described with reference to image acquisition device110including an X-ray imaging device. X-ray tube assembly112includes an x-ray tube generating x-ray energy to image a patient's breast tissue compressed between a support base and compression plate assembly116. Image acquisition device110can operate in different modes including standard mammography mode and tomosynthesis mode. Acquired images are stored in a data store or archive, e.g., a Picture Archive and Communication System (PACS) archive, and image files for acquired images are generated by image acquisition device110according to an imaging standard, one example of which is Digital Imaging and Communications in Medicine (DICOM). The DICOM standard provides a consistent platform for handling, storing, printing, and transmitting information in medical imaging and uses a specified file format and communications protocol.

FIG.1depicts an acquisition workstation120including one or more displays122(typically flat panel display that may include touchscreen functionality) through which an interactive UI124including one or more currently displayed images125is presented to a user or viewer (generally, user, such as technologist). Image acquisition workstation120also includes one or more input or peripheral devices127such as a keyboard, keypad, mouse, trackball, touchscreen, microphone for detection of voice commands, and a camera for video acquisition and user gesture or motion detection.

FIG.2depicts a breast image analysis system200including one or multiple image acquisition devices110and image review workstation120A in communication with each other through network130.FIG.2illustrates breast image analysis system200that may include one or multiple image acquisition devices110such as a conventional mammography acquisition device110a, a tomosynthesis acquisition device110b, an ultrasound acquisition device110c, a magnetic resonance imaging (MRI) acquisition device110d, and a generalized “other” medical imaging device110erepresentative of, for example, one or more computerized tomography (CT) imaging or positron emission tomography (PET) acquisition devices. Image review workstation120A may include one or more displays122a-b(generally display) with respective interactive UIs124a-cor output windows (generally, interactive UI) with which user can interact and that can be navigated using one or more user input or peripheral devices127. Image review workstation120A as depicted inFIG.2also includes an administrative display230for used for input and output of information associated with a particular set of images such as listings, tables, plots, text descriptions, etc. and system information. Image review workstation120A interactively displays the medial images to user in accordance with one or more UI programs carried out on an UI controller or processor240.

Breast image analysis system200may also include one or more image processing systems or programs in communication with network130. One example of processing system or program210for system generated data is a computer-aided detection (CAD) processor. A CAD processor that receives digital medical images from one or more image acquisition devices110to execute one or more CAD algorithms and generates system data in the form of CAD findings is provided. Another example of processing system or program210for system generated data is a post-Magnetic Resonance Imaging (MRI) processor for executing breast Dynamic Contrast Enhanced (DCE) colorization and providing kinetic findings (e.g., worst curve, post-processed by the system). While embodiments may involve different types of system generated data and associated processors or systems, reference is made generally to a system generated data generated by a CAD processor210as a non-limiting example.

Digital images acquired with one or more image acquisition devices110are stored to a data store or archive220(generally, data store), one example of which is a Picture Archiving and Communication System (PACS) archive. Data store220generally represents a repository for medical information associated with breast image analysis system200, including both current and archived images, current and archived system generated data such as CAD results, radiology reports for completed cases, and so forth.

Interactive UI124implemented at image review workstation120A interactively displays medical images125to a viewer or user via display122. Various medical images and related information are communicated according to the DICOM standard and network130supports the TCP/IP protocol, which is used as the transport protocol for the DICOM standard.

Further aspects of exemplary image acquisition devices110, image review workstations120A, data stores130such as PACS archives, and networked medical imaging environments are described in U.S. Publication No. 2017/0065238A1, published on Mar. 9, 2017, U.S. application Ser. No. 15/537,326, entitled “METHODS AND SYSTEMS FOR DYNAMICALLY MODIFYING ACQUISITION PARAMETER DURING IMAGE ACQUISITION,” which is commonly owned by Hologic, Inc., Marlborough, MA, and International Publication Number WO 2012/071429 A1, PCT Application No. US2011/061875, entitled “USER INTERFACE FOR MEDICAL IMAGING REVIEW WORKSTATION,” which is also owned by and names Hologic, Inc. as Applicant, the contents of which are also incorporated herein by reference as though set forth in full.

Breast image analysis systems constructed and operable according to embodiments and described in further detail with reference toFIGS.3-10include a snipping application incorporated into image review workflow presented by image review workstation120A during review of currently displayed image125and is seamlessly integrated into an existing medical imaging workflow. According to embodiments, image review workstation120A is triggered to automatically generate a digital image report or certain portion thereof based on pre-determined user interaction with interactive UI124. According to one embodiment, image review workstation120A is triggered to automatically generate a digital image report or a portion thereof in response to execution of snipping application to select a portion of currently displayed image125presented through interactive UI124, thus providing for automated, efficient and comprehensive digital image reports and complete or partial digital report generation while eliminating or reducing workflow interruptions during image review and troubleshooting.

FIG.3depicts an embodiment of a breast image analysis system300including image acquisition device110that generates breast tissue images310and corresponding images files320for acquired breast tissue images310. Image files320including image or image data310are stored in data store220such as PACS archive in communication with image acquisition device110and according to the DICOM standard (DICOM file or image file320inFIG.3). Data store220typically stores image files320for a multitude of patients. Image review workstation120A is in communication with data store220to read or access image files320and display images310through interactive UI124as described above with reference toFIGS.1-2. Image review workstation120A may also be in communication with a different or local data store330for storing copied or local image files310and/or images320. Image review station120A may also be in communication with a separate computing device340such as a computer of a user of another image review workstation120A.

According to embodiments, image review workstation120A includes computer-executable instructions of a snip-triggered digital image report generation application350(referred to as snipping application) stored in a non-transitory computer readable medium executable by a processor of image review workstation120A. Snipping application350is in communication with, utilized by or an executable component of UI controller240, which generates and modifies interactive UI124through display122of image review workstation120A such that UI controller240output results in presentation of currently displayed image125to user, and when snipping application350is executed, currently displayed image125is maintained or set as a background or working image by UI controller240. User is thus able to manipulate input or peripheral device127such as a touchscreen, keypad, keyboard, mouse or other input device to select or “snip” a portion125p(“p” referring to “portion”) of the currently displayed image125. Snipping or selection action executed by interactive UI124serves as a trigger to capture pertinent data and generate a digital image report360for that currently displayed image125or selected portion125pthereof. Digital image report360can then be transmitted as an electronic message to data store220/PACS archive, other data store330or computing device340.

Snipping application350may also be launched in response to other pre-determined user input based on voice recognition of a pre-determined verbal instruction spoken by user into input or peripheral device127in the form of a microphone of image review workstation120A or by detection of a pre-determined gesture or motion detected by input or peripheral device127in the form of a camera of image review workstation120A. Thus, embodiments not only provide for a “hands free” image snip and initiation of an image report, but post-snip actions such as verbal and video annotations may also be integrated into digital image report360.

User may also specify verbal dimensions for a snipped area or snipping application350may select a pre-determined snipping radius from a certain point such as a location of a mouse cursor over the currently displayed image. Accordingly, the shape and dimensions of selected portion125pof currently displayed image125may be drawn or specified by user or determined by image review workstation120A automatically or based on user-provided dimensions. For ease of explanation, reference is made to selected portion125pof currently displayed image125. Having described how system embodiments may be configured and system component operability, embodiments are described in further detail with reference toFIGS.4-10.

Referring toFIG.4, in a computer-implemented method400according to one embodiment, at402, image review workstation120A presents interactive UI124with currently displayed image125of breast tissue for a patient through display122of image review workstation120A. At404, image review workstation120A receives user input to execute snipping application350. At406, image review workstation120A, in response to the trigger of execution of snipping application350, automatically generates digital image report360for snipped or selected portion125pof currently displayed image125.

Referring toFIG.5, breast image analysis system500or medical imaging environment configured according to one embodiment includes one or more image acquisition devices110in communication with image review workstation120A, display122through which interactive UI124is displayed and with which user interacts, input devices127such as a mouse, keyboard, keypad, touchscreen, microphone or camera as discussed above, and data store220such as PACS archive, which stores image files320generated by image acquisition device110according to the DICOM standard. While embodiments are described with reference to DICOM standard and DICOM image files320, it will be understood that embodiments are not so limited and that embodiments may be implemented using various image file standards.

In the illustrated embodiment, image review workstation120A includes UI controller240for processing and displaying interactive UI124and user interactions with interactive UI124and snipping application350is in communication with UI controller240or a component of UI controller240. In one embodiment, snipping application350is structured to include a snip function510, an aggregator520and a report generator530, which outputs digital image report360. In other embodiments, snipping application350includes additional elements for other digital image report360capabilities and configurations including metadata import540and configuration file550. In other embodiments, snipping application350also includes additional image identifier560. Snipping application350may also include system generated data import to integrate system generated data210such as data generated by CAD processor or post-MRI processor for executing breast DCE colorization and providing kinetic findings. For these embodiments, system generated data sources such as CAD processor210also provide system generated data to aggregator520. Other embodiments of snipping application350may import data from various combinations of sources noted above. Thus, the system configuration shown inFIG.5is provided as one example of how embodiments may be implemented. Individual components of snipping application350, their operability and interoperability are described in further detail below.

Snip function510provides for snipping or selection of portion125pof currently displayed image125. Snip function510may output selected portion125pin the image format of image file320standard (such as DICOM), or output generated by snip function510in the form of snipped or selected portion125pof currently displayed image125. Snip function510output may be provided as an input to a converter512to generate selected portion125in a different image format, e.g., to convert selected portion125pfrom a DICOM format to a .jpeg format or other image format if digital image report360is to be generated with images of a certain format. While embodiments are described with reference to report360in an “image” format, embodiments may involve a generated report360that includes portions of multiple types or formats (image and non-image), and report360may also be a non-image format such as text, audio, video, JavaScript Object Notation (JSON), DICOM or Health Level 7 (HL7) format. Thus, “converter”512is not limited to an image format converter.

Snip function510is also in communication with aggregator520, which is configured to collect images and data that are to be captured and incorporated into digital image report360to be generated. In the illustrated embodiment, aggregator520receives output or selected portion125pgenerated by snip function510.

Snipping application350may also include metadata import540that includes or reads configuration file550including pointers to or identifiers of certain portions of image files320in data store220. In the illustrated embodiment, configuration file550includes a first identifier551utilized to locate image file320in data store220containing or corresponding to currently displayed image125, and a second identifier552or additional identifiers of elements within image file320in the form of tags571utilized to identify and read certain metadata572of image file320. Image file320may include a large number of tags571and associated metadata572related to, for example, patient demographic data, study and series data, and image acquisition data. Configuration file550is used to specify only a subset of image file tags571contained in image file320. Tags571of configuration file550may be pre-set by the manufacturer of image review workstation or selected by the user of image review workstation120A. Metadata import540is configured to communicate with data store220, locate and access image file320stored in data store220and related to currently displayed image125, and provide any metadata572identified using tags571retrieved from image file320to aggregator520.

Aggregator520serves as a data collection element, or a type of temporary “clipboard,” which provides collected data including selected portion125pof currently displayed image, converted512versions thereof and any metadata572read from image file320to report generator530. Aggregator520may also collect system generated data such as data generated by CAD processor210or post MRI processor.

Report generator530is configured to access an image report template531and populate image report template531. According to one embodiment, template includes a section to be populated with selected portion125pand other data collected by aggregator520and received by report generator530from aggregator520. For this purpose, template531may have pre-defined fields or sections to be populated with images, pre-defined fields or sections to be populated with other data such as image file metadata172and system generated data.

In certain embodiments, snipping application350, by additional image identifier550, may also locate other images552related to or associated with currently displayed image125or selected portion125pthereof, e.g., other images for the same patient and in a different series of images acquired on different dates during the course of imaging analysis and treatments. These other related images552may also be provided to aggregator520so that the digital image report360that is generated may include one or more or all of the snipped or selected portion125pof currently displayed image125, currently displayed image125itself, and one or more additional related images552. Resulting image report360may also include retrieved metadata572for currently displayed image125and/or for other related images552and related system generated data (e.g., CAD and post MRI data) related to same. Further details of one embodiment of how a breast analysis system including an image acquisition device110and image review workstation120A executing snipping application350operate are described with reference toFIGS.6-10.

Referring toFIG.6, at602, patient breast tissue is positioned or compressed by components of image acquisition device110, such as a tomosynthesis imaging device, which generates digital images310of patient's breast tissue. At604, image acquisition device110also generates respective digital image files320(e.g., DICOM files) for respective acquired breast tissue images/image data310or sets or series of images310. This process may be repeated for different patients and for different series or dates for a given patient. At606, image acquisition device110stores respective digital image files320containing images310or image310data to data store220such as PACS archive that is also accessible by image acquisition device220.

According to one embodiment, image files320are structured according to DICOM standard.FIG.7Agenerally depicts how DICOM image file320generated by image acquisition device310is structured by grouping information into pre-defined data sets. As shown inFIG.7A, image file320in the form of DICOM file includes a header710and image data720. With further reference toFIG.7B, header710contains various types of information about a breast imaging procedure including demographic or patient information, imaging parameters, and image acquisition device data. Header710is followed by intensity data720for the pixels of the image (binary 1s and 0s) and this data can be used to reconstruct an image or frames thereof.

Metadata572of DICOM image file320is referenced by respective tags571, which serve as alpha-numeric pointers or references to corresponding metadata572. For example, a DICOM header710may include identification or demographic information such as Patient ID referenced by tag (0010,0020), Patient Name referenced by tag (0010,0010). Header710may also include patient demographic information such as Patient's Age referenced by tag (0010,0101), Patient's sex referenced by tag (0010,0040). Header710may also include information about image acquisition device110, image acquisition device110configuration, image attributes, and the imaging study being conduct. For example, header710may include information about acquisition date referenced by tag (0008,0022), type or format of data format utilized, e.g., video image format acquired by tag (0008, 1022) or digital image format acquired by tag (0008, 0123), and attributes or operating parameters of different types of image acquisition devices110, e.g., whether MRI, CT, X-ray, or tomosynthesis. The DICOM standard uses hundreds of tags571referencing respective metadata572about the patient, images, image acquisition device, operating parameters, demographics, etc. While the types of DICOM tags571are comprehensive, the user of the image review workstation120A is typically interested in only a subset, often a small subset, of available header710data. Image file310in form of a DICOM file also include a data set for image pixel intensity data573, which can be used to reconstruct an image.

Referring toFIG.8, following image acquisition device110acquiring images310and generating image files320, image review workstation120A receives a user request through interactive UI124to view subject's breast tissue image310at802, and at804, image review workstation120A communicates with data store220. At806, image review workstation120A reads a first or image file identifier of user request to locate the image file310in data store220, and at808, image review workstation120A accesses or receives image file320or associated image data720to reconstruct requested image.

At810, and with further reference toFIG.9A, image review workstation120A, by UI controller240, presents interactive UI124including the accessed or received breast tissue image, as generally depicted inFIG.9A. In the illustrated example, interactive UI124includes currently displayed image125of breast tissue, navigation or workflow tools902(e.g., zoom in, zoom out, next image, prior image, next series, prior series, optical adjustments such as brightness, contrast, etc.) Interactive UI124may also include imaging, acquisition and lesion attribute data or UI elements to display such data, examples of which include capture markings and annotations, lesion measurements (size of lesion, including orthogonal measurements), lesion location (e.g., determined with reference to a ruler or other location/distance measurement, including for left and right breast, quadrant, o'clock region), and relative lesion distance information of lesion such as distance of a lesion from a chest wall or nipple of breast. User interacts with interactive UI124, and at812, image review workstation120A, by UI controller240, processes user navigation of interactive UI124and currently displayed image125. As part of the review workflow, user may review various patient images, different image views, and images acquired using different imaging modalities.

In the illustrated embodiment, interactive UI124also includes an icon904for snipping application350, which may be activated by user clicking on icon904with a mouse or by other input device127, e.g., with a pre-determined key of a keyboard, a pre-determined verbal instruction spoken by user into microphone of image review workstation120A, or by a pre-determined gesture detected by camera of image review workstation. For example, the user may select “Alt-S,” (“s” for “snip”), click a certain button on a mouse, speak a pre-determined phrase such as “activate snip” into microphone or make a pre-determined gesture such as waving their hand in front of the camera or other pre-determined gesture to request execution of snipping application350. For ease of explanation, reference is made generally to a snipping application350and activation of same based on a pre-determined user input.

Continuing with reference toFIG.8, at814, user has submitted input to request to activate snipping application350, which is received by UI controller240. At816, and with further reference toFIG.9B, processor of image review workstation120A executes computer-executable instructions of snipping application350stored in non-tangible computer readable medium or read into memory in response to user input, and at818, UI controller240, by snip function510as shown inFIG.9B, selects or crops portion of currently displayed image125of subject's breast tissue based on user input.

According to one embodiment, as generally depicted inFIG.9B, selection or snipping is executed based on user manipulation of a mouse, touchscreen, keyboard, keypad or other input device127that allows a user to define a snip area of selected portion125pof currently displayed image125. Selected portion125pmay include only a selected segment of the displayed tissue, or in other words, be defined within a boundary of tissue of currently displayed image125as illustrated inFIG.9B, or selected portion125pmay encompass the entire displayed tissue. In other embodiments, snip function510automatically captures the entire displayed screen of interactive UI124without requiring user to define snip area. In yet other embodiments, snip function510captures a pre-defined area defined by a pre-defined radius or predefined vertical and horizontal distance from a pre-determined point. For example, pre-determined point may be a center of currently displayed image125or a current location of mouse pointer such that snip function510determines, e.g., with reference to ruler906measurements, e.g., 1 inch in each direction from mouse cursor to snip a square shape or snip a circular shape with a 1 inch radius. For ease of explanation, not limitation, reference is made to embodiments in which user defines snip area by manipulating mouse device127to define a selected portion of the currently displayed image125. Referring again toFIG.5, selected portion of currently displayed image is provided by snip function510to aggregator520, which collects data to be utilized for digital image report360generation.

Referring again toFIGS.5and8, at820, if necessary, image review workstation120A converts512snipped or selected portion125pof currently displayed image125as necessary. For this purpose, as depicted inFIG.5, output of snip function510in the form of selected portion125pof currently displayed image125may be converted from a first image format a second image format, e.g., from a DICOM format to a .TIFF format, a .JPEG format, a .PNG format, or a .GIF format. Converter512may also be or include a non-image converter to generate audio, video, JSON or DICOM file as necessary, e.g., for capture of .AVI file for multiple image frames or a video of the interactive UI124to record user interactions interactive UI124to allow radiologist to review a video, scroll through slices or images of a slab or to review patient movement for MRI motion correction. Reference is made generally to converter512and embodiments may thus utilize the original DICOM format of the currently displayed image125without conversion, or convert512selected portion125pof currently displayed image125from first image format to a different, second image format more for digital image report360. Selected portion125pof currently displayed image125, whether converted or not converted, is provided by snip function510to aggregator520.

With continuing reference toFIGS.5and8, at822, image review workstation120A may receive real-time user annotations relating to currently displayed image125or selected portion thereof125pvia one or more of interactive UI124and input or peripheral device127if applicable, and these user provided annotations are also provided to aggregator520. For example, user may enter notes, comments or observations via keyboard or touchscreen, and these annotations are received via interactive UI124and provided to aggregator520. As another example, user may speak notes, comments or observations into microphone of image review workstation120A, and the resulting audio file or text file resulting from audio to text conversion are provided to aggregator520. As yet another example, image review workstation120A may receive user annotations based on user input via a computer mouse, e.g., highlighting a certain part of captured screen or image portion or to point out certain data such as image, acquisition or demographic attributes or data.

It will be understood that user annotations may or may not needed and that user annotations may concern various aspects of the currently displayed image125, patient, acquisition parameters, etc. For example, the user may provide comments or notes regarding the currently displayed breast tissue such as the amount of Fibroglandular Tissue (FGT), the amount and symmetry of background Parenchymal Enhancement (BPE), an observed lesion type or lesion attributes such as focus, mass, non-mass enhancement, skin lesion, intramammary Lymph Node, an observed fat-containing lesion or attribute such as lymph nodes and whether what is observed is normal, non-enhancing findings such as a cyst or non-enhancing mass, associated features such as architectural distortion, and whether any implants are observed.

Continuing with reference toFIGS.5and8, in certain embodiments that also read metadata572from the acquired image file320for inclusion in digital image report360to be generated, image review workstation120A determines identifiers or tags571of acquired image file320at824. For this purpose, in the illustrated embodiment, snipping application350includes metadata import540that includes or reads configuration file550that specifies DICOM tags571used to read metadata572be read and provided to aggregator520. In one embodiment, DICOM tags571can be specified or selected by the user before the user begins an image review workflow. In another embodiment, DICOM tags571are specified or selected by the user in real time during an image review workflow via interactive UI124. User may review a list of possible DICOM tags571or tag categories (as shown inFIG.7B) that can be selected. A DICOM image file320includes hundreds tags571for corresponding metadata572, and configuration file550specifies a subset of available tags571for metadata572that is most pertinent or of interest to the image review workstation user or person who will be receiving the digital image report360. Digital image report template531may also be configured to accommodate a limited number of types of metadata572. In this manner, digital image report360is not cluttered with extraneous metadata572that is not needed or of interest.

In embodiments involving DICOM image files320, configuration file550may include a list of DICOM tags571structured as eight alphanumeric elements (xxxx,yyyy) for identifying metadata572to be read from an acquired image file320associated with currently displayed image125. For example, a first metadata tag571or first group of metadata tags571may be used to identify acquired image file320in data store220that was generated for currently displayed image125if it is necessary to access or retrieve acquired image file320that is the source of currently displayed image125, e.g., if image review workstation120A does not have a local copy of acquired image file320. For this purpose, metadata import540may read a unique file identifier of currently displayed image125and locate corresponding identification data in data store220to identify corresponding acquired image file320, or identify matching metadata572using tags571that can be used for unique identification such as one or more of Patient Name (0010,0010), Patient ID (0010,0020), Patient Birthdate (0010,0030), Patient Address (0010,1040), Patient Telephone Numbers (0010,2154), specified in configuration file550.

As another example of how one embodiment may be implemented, configuration file550may identify a subset of DICOM header tags571for patient/demographic data such as patient name, data of birth and sex, study and series level information such as study date; study/series description, institution name, image acquisition device manufacturer, model and imaging modality, laterality, body part examined; and image level information such as image acquisition parameter data such as kilovolt peak (kVp) data, exposure data (mAs), dose data (ASD, ESD, EI), compression force, body part thickness, x-ray detector data, and filter data. It will be understood that configuration file550may specify various tags571for different types of metadata572, and that the exemplary subset provided above is illustrative of types of DICOM tags571and associated metadata572that may be included in digital image report360and so that digital image report360content can be customized by the user of image review workstation120A.

It will be understood that an acquired image file320corresponding to currently displayed image125can be identified using various unique identifiers, identifier tags and combinations thereof to ensure that the correct acquired image file320for currently displayed image125is identified, e.g., if multiple acquired image files320for a patient have been generated for prior diagnosis or treatments. Further, it will be understood that image review workstation120A may utilize acquired image file320that is copied and stored locally at image review workstation120A or image review workstation120A may access acquired image file320stored in data store220.

With continuing reference toFIG.8, and referring again toFIG.7B, at828, image review workstation120A accesses acquired image file320associated with currently displayed image125and, identifies tags571of configuration file550, and reads respective metadata572associated with respective tags571. Metadata572read from acquired image file320is provided by metadata import540to aggregator520. At830, and referring again toFIG.5, image review workstation120A automatically generates digital image report360incorporating the data collected. For this purpose, aggregator520provides data collected or retrieved from snip function510, metadata import540, image converter512, user annotations and/or system generated data from CAD processor or post-MRI processor210as examples. Report generator530populates respective portions of image report template531with respective image(s), converted images, user annotations, meta data and/or system generated data as appropriate.

Thus, resulting digital image report360may include one or more or all of the data elements collected by aggregator520. For example, according to one embodiment, digital image report360includes only the selected portion125pof currently displayed image125(in original DICOM format or in another image format after image conversion512). According to another embodiment, digital image report360includes selected portion125pof currently displayed image125and metadata572read from corresponding image file320. According to a further embodiment, digital image report360includes selected portion125pof currently displayed image125and user annotations or notes (whether recorded as text entered via keyboard, touchscreen, mouse highlights or notations, an audio file generated by user speaking into microphone for annotations, text resulting from conversion of the audio file, or a video file of the user's interactions with the interactive UI124). According to another embodiment, digital image report360includes selected portion125pof currently displayed image125, user annotations and metadata571of corresponding acquired image file320. According to other embodiments, digital image report360includes system generated data such as CAD or kinetics findings generated by CAD or post MRI processor210.

Thus, it will be understood that generated image report360may integrate various types eps of data received from aggregator520, and that the resulting image report360may or may not include metadata571read from image file320. It will also be understood that digital image report360and components thereof provide by aggregator520may be in various formats depending on the desired output format of digital image report360and content thereof. For example, digital image report360or components thereof may be in a standard format such as a .TIFF format, a JPEG format, a .PNG format, a .GIF format or a non-image format such as audio format or video format such as .AVI. Further, digital image report360or components thereof may be structured according to non-image formats such as DICOM format or series (e.g., derived series with Referenced Service-Object Pair (SOP) class, Instance Unique Identifiers (UID) and HL7. Digital image report360may also be configured as a text file for cases in which digital image report360includes information about an image but not an image itself, or portions of digital image report360may be in the form of a text file. In another embodiment, digital image report360may be in the form of a clipboard structure that serves as a temporary file or storage such that the data of digital image file can be copied to a Word or PowerPoint file. Digital image report360may also be configured as a delimited text file (e.g., based on National Mammography Database (NMD) file specifications, which may also be exported and submitted to NMD or other reporting system. Digital image report360may also be structured according to a user or manufacturer defined container format, which may be open to the public and that can be integrated with external reporting and dictation systems. Accordingly, it will be understood that digital image report360or particular sections or content thereof may be structured in different ways, and that digital image report360may be structured according to a single or multiple data formats.

With continuing reference toFIGS.5and8, at832, image review workstation120A may execute one or more post image report generation actions such as presenting digital image report360through display122of image review workstation120A, storing digital image report360in local data store330, transmitting digital image report360to data store220or PACS archive, and for this purpose, digital image report360may be stored together with or linked to corresponding acquired image file(s)320. Image review workstation120A may also transmit digital image report360through communication network130to user computer340for subsequent review after the image review workflow with image review workstation120A has been completed. The user computer340may be connected to the data store220through the communication network130.

In some embodiments, the user computer340is another image review workstation similar to the workstation120A, where the user may either retrieve the digital image report360via data store220or PACS archive or may directly receive the digital image report360through communication network130. A user of the user computer340may include a radiologist. The radiologist may access the digital image report360and the other image review workstation may access the metadata572and the image file320, including any tags or annotations. The radiologist may then validate the conclusion of the first user or radiologist reached in the first review. The radiologist may then access the images described and included in the digital image report360for validation and conclusion or use the information for further diagnosis or treatment.

In other embodiments, the user computer340may be a computer system accessed by a technologist for further diagnostic imaging. The user computer system may be similar to the acquisition workstation120shown inFIG.1. The technologist may acquire additional images based on the information and images contained in the digital image report360.

In additional embodiments, the user computer340may be part of a patient reporting system and may use the information contained in the digital image report360to generate reports provided to a patient imaged by systems110a-e.

While certain embodiments have been described above with reference to snip-triggered generation of a digital image report360including a selected portion125pof a currently displayed image125, other embodiments involve incorporating multiple selected portions125pof currently displayed image125. Yet other embodiments involve incorporating additional images or additional image portions, or other views thereof, into the digital image report360such that the digital image report360includes at least two images or portions thereof. Such digital image reports360may also include one or more or all of metadata372from one or more or all of respective image files320, user annotations, converted images and system generated data. Embodiments involving multiple or additional image captures allow for consolidation of multiple captures into a single digital image report360for even more comprehensive assessment that may involve, for example, patient level assessment, left/right breast assessment, fibroglandular tissue (FGT) and background parenchymal enhancement (BPE) assessment, and assessments of a combination of different procedures or results of different imaging modalities and results generated thereby.

For these purposes, currently displayed image125may include a reference to other images or other image views, e.g., as stored in the data store220. Thus, while a portion125pof currently displayed image125is selected or snipped, snipping application350executes to locate other related images or other views of the currently displayed image or portion thereof.

Referring again toFIG.5, breast image analysis system or medical imaging environment constructed according to one embodiment includes components described above with reference toFIG.5and further provides for additional image retrieval560. Additional images562to be retrieved can be identified by database564of acquired images, including the currently displayed image125, and references to other images562to be retrieved. Further aspects of embodiments are described with reference toFIG.10, and various system and operability details described above are not repeated.

Referring toFIG.10, at1002, image review workstation120A executes computer-executable instructions of snipping application350in response to user input, and a portion125pof currently displayed image125of subject's breast tissue snipped or selected. At1004, snipping application350identifies one or more additional images562associated with the currently displayed image125. The one or more additional images562may be additional images of the same view (e.g., image of same view for same patient at an earlier date to provide for image comparison), additional images562of different views of the currently displayed image or different views of the selected portion of the currently displayed image125to provide a different visual perspective of the selected portion of the currently displayed image of interest, or a combination thereof. For this purpose, additional image retrieval560may maintain or access a database564cross referencing images and related views of the currently displayed image125, which may be in the same data store or archive or different data stores. At1006, snipping application350, using data store pointers or identifiers, address data and/or related image identifiers in database564, accesses the referenced data store to access or retrieve additional images562at1008, and at1010, image review workstation120A reads or makes a copy of the additional image file related to the currently displayed image125. At1012, image review station120A may also access configuration file550to determine DICOM tags571to retrieve metadata572about the additional images562or image views, and at1014, performs image conversion512as necessary. Additional images562and any metadata572and converted512images are provided to aggregator520such that aggregator520includes selected portion125pof currently displayed image125and at least one additional image562related to currently displayed image125, an image562of a different view of currently displayed image125and an image562of a different view of selected portion125pof currently displayed image125. Image562of different view of selected portion125pmay be identified, selected and aligned with reference to ruler measurements906, dimensions and distance data of selected portion125pof currently displayed image125. Such measurements, dimensions and distance data can be used in another image or other view562to identify selected portion125pin the other image or view562. Aggregator520may also include one or more or all of converted images, user annotations and metadata572concerning the selected portion125pof currently displayed image125and/or one or more or all of the additional images562. At1014, image review station120A automatically generates digital image report360including selected portion125pof currently displayed image125and at least one additional image562, and in certain embodiments, associated selected metadata572.

For example, currently displayed image125is presented through interactive UI124in a first view such as an axial view of selected portion125p. Image review workstation120A identifies selected portion125pof currently displayed image125using one or more of interface ruler906, measurements or relative dimensions in a second view (or third or other view). For example, the first view may be an axial view of the selected portion of the currently displayed image, and the second view may be a sagittal view or a coronal view of the selected portion of the currently displayed image. In cases in which a third view is utilized, the second view is one of the sagittal view and the coronal view, and the third view is the other of the sagittal and coronal view.

As another example of how embodiments may be implemented, the second view is a computer-generated rendering of selected portion125pof currently displayed image125such as a maximum intensity projection rendering, multiplanar reconstruction rendering, a subtraction rendering, a CAD rendering or an MRI processed output such as a rendering of breast DCE decolorization. As a further example, a first view may be based on an acquired optical image of the subject's breast tissue, and a second view is a computer-generated rendering of the subject's breast tissue.

Embodiments involve additional images562acquired by the same acquisition device110and modality, which may also be part of the same image file320. In another embodiment, additional image562is acquired by a different acquisition device110using a different imaging modality, and additional image562is part of a different image file320. For example, currently displayed image125may have been generated by a first image acquisition device110in the form of MRI device, whereas an additional image562was generated by a second image acquisition device110such as one of a tomosynthesis imaging device, an x-ray imaging device, or an ultrasound imaging device.

Thus, it will be understood that embodiments may identify additional images562of the same type, of different types, acquired using the same device/modality, different devices/modalities, and that the additional images562may be images of the currently displayed image125or selected portion125pthereof, as well as different views thereof, and that these additional images562can be provided to aggregator520for incorporation into digital image report360to be generated.

Embodiments thus provide for efficient and automated digital image report generation that is in-line with a current review workflow while also providing for more comprehensive image reports to reduce errors and provide additional image data as a result of being able to incorporate one or more or all of images, metadata, system generated data and user annotations automatically into a single digital image report while eliminating or reducing the need for manual entry of such data which can be very cumbersome and interrupts the user's workflow. Embodiments also provide for more effective tracking of tumor response criteria, e.g., as set forth in Response Evaluation Criteria in Solid Tumors (RECIST), which is a widely accepted standardized measure of tumor response and establishes definitions for minimum size of measurable lesions, instructions about how many lesions to follow, and the use of unidimensional measures for tumor evaluation.

Embodiments also enhance engineer image reviews when attempting to identify and resolve software issues related to how images based on the same image acquisition data appear differently when using different software programs and thus enhancing troubleshooting the sources of image inconsistencies. Embodiments also enhance radiologist review and workflows by providing for the ability to not only request generation of a digital image report during a workflow, but also being able to specify the digital image content report for a more comprehensive summary without being interrupted or having to utilize a different computing system to prepare a report or record notes and observations.

While certain embodiments have been described individually, other embodiments may involve a combination of those individually described embodiments.

For example, while certain embodiments have been described with reference to generation of a digital image report based on an image snip, it will be understood that embodiments may involve generation of a complete digital image report or partially populating or generating a digital image report. As another example, while certain embodiments have been described with reference to generation of a digital image report in certain exemplary formats, it will be understood that other image formats may be utilized, and that a report may also be in, or contain portions in, a non-image format such as text, video, JSON or DICOM, HL7 and proprietary formats. Moreover, while certain embodiments are described with reference to a generated image report including a selected portion of an image, it will be understood that a generated image report may include one or more or all of the currently displayed image, the selected portion of the currently displayed image, one or more other related images or selected portions thereof that are not displayed to the user. Additionally, while certain embodiments are described with reference to snip triggered generation of a digital report and associated images and image metadata, it will be understood that embodiments may involve generation of a complete or partial digital report in a certain format including one or more or all of a currently displayed image or selected portion thereof (or images and portions thereof), another image that is not currently displayed or certain portion thereof, and associated data including one or more or all of image metadata (patient, study, series, image), user input or annotations (such as spoken annotations and measurements), and system generated input (e.g. kinetics, CAD data), and that the report format may be an image or non-image format.

While multiple embodiments and variations of aspects of the invention have been disclosed herein, such disclosure is provided for purposes of illustration only. Where methods and steps described above indicate certain events occurring in certain order, those of ordinary skill in the art having the benefit of this disclosure would recognize that the ordering of certain steps may be modified and that such modifications are in accordance with the variations of the invention. Additionally, certain of the steps may be performed concurrently in a parallel process as well as performed sequentially. Thus, the methods shown in various flow diagrams and described with reference to particular snipping application configurations and execution are not intended to be limited to a particular sequential order, particularly in instances in which certain steps may or may not be performed. Accordingly, embodiments are intended to exemplify alternatives, modifications, and equivalents that may fall within the scope of the claims.