VIEWING AND CORRELATING BETWEEN BREAST ULTRASOUND AND MAMMOGRAM OR BREAST TOMOSYNTHESIS IMAGES

This specification describes a novel user interface and method for viewing mammograms together with 3D breast ultrasound images for breast cancer screening. The user identifies a region of interest, in one modality and the processing system calculates the location in the other modality. Visual aids such as icons can be used to display the calculated location in the other modality in cases where the other modality is viewed on a separate device. In cases where an integrated display device is used that displays both modalities, the visual aid can be an ROI marker on the target modality. The user interface and method for viewing has been found to greatly decrease the tedium and likelihood for errors when compared with known displaying viewing techniques.

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments of the present invention. Those of ordinary skill in the art will realize that these various embodiments of the present invention are illustrative only and are not intended to be limiting in any way. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure.

In addition, for clarity purposes, not all of the routine features of the embodiments described herein are shown or described. One of ordinary skill in the art would readily appreciate that in the development of any such actual embodiment, numerous embodiment-specific decisions may be required to achieve specific design objectives. These design objectives will vary from one embodiment to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine engineering undertaking for those of ordinary skill in the art having the benefit of this disclosure.

This specification describes a novel user interface and method for viewing mammograms together with 3D breast ultrasound images for breast cancer screening. The user interface and method for viewing has been found to greatly decrease the tedium and likelihood for errors when compared with known displaying viewing techniques. The techniques can be used in separate displays, such as shown inFIG. 1Bor1C, or according to some embodiments, can be used in an integrated display such as integrated device150shown inFIG. 1D.

FIG. 1Bis a diagram illustrating a system configured to create and display visual aids for indicating locations on mammogram images that correspond to selected ROIs on ultrasound images when separate viewing devices are used, according to some embodiments. Device130has a display screen131used to display 2D ultrasound breast images to a user. Device130also includes input devices such as keyboard and mouse, and a processing system134. According to some embodiments, other user input methods such as touch sensitive screen screens can be used. Processing system134can be a suitable personal computer or a workstation that includes one or more processing units136, input/output devices such as CD and/or DVD drives, internal storage138such as RAM, PROM, EPROM, and magnetic type storage media such as one or more hard disks for storing the medical images and related databases and other information, as well as graphics processors suitable to power the graphics being displayed on display131. As will described in further detail herein, when a user, with our without the use of computer aided diagnosis (CAD), selects an ROI on the ultrasound images displayed on display131, the processing system134automatically calculates coordinates in a mammogram that correspond to the location of the selected ROI. According to some embodiments, a visual aid132in the form of icons are automatically presented to the user on screen131so that the user can quickly and efficiently find the corresponding location on a separate device140being used to display the mammogram images for the same patient.

FIG. 1Cis a diagram illustrating a system configured to create and display visual aids for indicating locations on ultrasound images that correspond to selected ROIs on mammogram images when separate viewing devices are used, according to some embodiments. Device140has a display screen141used to display mammogram breast images to a user. Device140also includes input devices such as keyboard and mouse, and a processing system144. According to some embodiments, other user input methods such as touch sensitive screens can be used. Processing system144can be a suitable personal computer or a workstation that includes one or more processing units146, input/output devices such as CD and/or DVD drives, internal storage148such as RAM, PROM, EPROM, and magnetic type storage media such as one or more hard disks for storing the medical images and related databases and other information, as well as graphics processors suitable to power the graphics being displayed on display141. As will described in further detail herein, when a user, with our without the use of CAD, selects an ROIs on the mammogram images displayed on display141, the processing system144automatically calculates coordinates in a 3D ultrasound image that correspond to the location of the selected ROI. According to some embodiments, a visual aid142in the form of icons are automatically presented to the user on screen141so that the user can quickly and efficiently find the corresponding location on a separate device130being used to display 2D ultrasound images for the same patient. According to some embodiments, the systems130and140ofFIGS. 1B and 1Ccan both be used in combination such that the user can quickly and easily find an ROI location in one modality upon selecting an ROI in the other modality.

FIG. 1Dis a diagram illustrating a system configured to create and display visual aids indicating locations on images of one modality that correspond to selected ROIs on another modality when using a device that is configured to simultaneously display images of both modalities, according to some embodiments. Device150has a display screen151used to display both mammogram breast images as well as 2D ultrasound images to a user. Device150also includes input devices such as keyboard and mouse, and a processing system154. According to some embodiments, other user input methods such as touch sensitive screens can be used. Processing system154can be a suitable personal computer or a workstation that includes one or more processing units156, input/output devices such as CD and/or DVD drives, internal storage158such as RAM, PROM, EPROM, and magnetic type storage media such as one or more hard disks for storing the medical images and related databases and other information, as well as graphics processors suitable to power the graphics being displayed on display151. As will described in further detail herein, when a user selects an ROIs on the mammogram images displayed on display151, the processing system154automatically calculates coordinates in a 3D ultrasound image that correspond to the location of the selected ROI. According to some embodiments, visual aids in the form of ROI markers are automatically presented to the user on overlaid on the appropriate 2D ultrasound images. Similarly, according to some embodiments, when the user selects and ROI on the 2D ultrasound image(s), the processing system154automatically calculates the coordinates of the ROI on one or more mammogram images, and automatically displays the coordinates, such as in form of ROI markers overlaid directly on the mammogram images being displayed on display151. In this way when a user selects an ROI in one modality, the user can quickly and efficiently find the corresponding location on the other modality.

The described systems can be configured to automatically calculate the coordinates on the corresponding mammograms of an ROI found on 3D ultrasound images and to present roadmaps indicating the location on the mammograms and vice versa. According to some embodiments, a roadmap can be constructed to indicate the corresponding location of an ROI found on ultrasound in sketches of the mammogram standard views, CC and MLO views for example.FIGS. 2A-2Care diagrams illustrating a roadmap created using sketches of mammograms and ultrasound views, according to some embodiments.FIG. 2Ashows a coronal view icon210with a ROI212.FIGS. 2B and 2Cshow CC view and MLO view icons220and230respectively. Breast roadmap icons220and230show the ROI location markers222and232, along with a nipple marker such as marker234in icon230.

According to some embodiments, the breast icons such as shown inFIGS. 2A-2Care used as a roadmap to indicate the corresponding locations of the ROI found in another modality.FIGS. 2D-2Fillustrate aspects of a roadmap created using down sampled images, according to some embodiments. As an alternative to the sketches as shown inFIGS. 2A-2C, miniatures (down sampled image) of the mammogram or of the coronal view of the ultrasound shown can be used for the roadmap. Images240,250and260are down samples images of coronal view, CC view and MLO view images respectively. On the icon images240,250and260are shown ROI markers242,252and262respectively, as well as nipple markers such as marker254in icon image250. In general, the icon can be displayed with a marker (dashed circle in the figure) to indicate the location of the ROI. According to some embodiments, the coordinates of the ROI, such as nipple distance, angle, clock face position and distance from the skin (i.e. from the compression paddle for mammogram and tomosynthesis and from the probe surface for ultrasound), etc. is also displayed together with the icons, as shown inFIGS. 2A-2F.FIGS. 3A-3Care diagrams illustrating a definition of location coordinates, according to some embodiments. Diagrams310,320and330illustrate example definitions for clock position, angle, nipple distance and skin distance in breast icons, according to some embodiments.

FIG. 4illustrates a 3D ultrasound display device with breast icons, according to some embodiments. Display screen400corresponds to screen131of device130shown inFIG. 1B. Screen400includes a coronal view410shown displaying a current slice at a given depth, along with the two common orthogonal views, namely sagittal view420and transversal view430. A nipple marker414is also shown on coronal view410. The user can move around any of the three user controlled cursors412,422and432on views410,420and430respectively. When the user moves one of the cursors, the display device automatically calculates the corresponding image slice and location of the other two cursors for the other two views. According to some embodiments, when the user moves one of the user controlled cursors412,422or432to survey the ultrasound images, the ROI indicator, shown as dashed circle such as indicator442, in the breast icons450,452and545move accordingly to indicate the location of the ROI on the corresponding mammograms. Also shown in the breast icons are nipple markers, such as nipple marker460. Thus, in the case of the user is viewing mammogram images on a separate monitor, the user can conveniently use the displayed ROI indicators on the icons452and454to more quickly identify the corresponding location on the mammogram images. This has been found to greatly reduce the tedium associated with trying to locate the corresponding location completely manually. Furthermore the likelihood of making errors when compared to the manual method is greatly reduced.

FIG. 5illustrates a mammogram display device with breast icons, according to some embodiments. The display500corresponds to screen141of device140shown inFIG. 1C. Screen500includes CC views510and MLO views520. The user can move the user defined cursors512and522on vies510and520respectively. To find the ROI on ultrasound from a mammogram finding, the user identifies the finding or lesion on at least two views of the mammograms. The location of the ROI on the ultrasound images can then be calculated and displayed on the ultrasound views accordingly. In the case of breast tomosynthesis, the user needs only to identify the lesion on one of the views and the location of the ROI can be calculated and displayed on the ultrasound. In the case ofFIG. 5the location of the ROI is displayed on the icons540including a dashed circle ROI marker on the coronal view icon as shown. In the case where the user is viewing a 3D ultrasound image on a separate monitor, the user can more quickly and conveniently locate the ROI on the 2D ultrasound image views. In particular, the user knows the approximate location based on the icon (as well as on the clockface, nipple distance and skin distance numbers displayed under the icon). The user can then scroll through the coronal slice images to locate coronal slice that corresponds to the identified ROI. As in the case of ultrasound to mammography, the case of mammography to ultrasound greatly reduces both the tedium and likelihood of errors when compared to a purely manual method.

FIG. 6. illustrates an integrated display device that is configured to display both mammogram and ultrasound images and to automatically calculate corresponding locations, according to some embodiments. For an integrated mammogram and ultrasound-viewing device, the cross-modality ROI correlation is more straightforward. The location of the ROI found in one modality image is directly marked in the images of other modality.

When CAD is available to one or both modalities, the ROI could be generated by CAD with or without a physician double checking it first. Or the ROI could be generated by the physician and CAD is used as a second-reader. Whenever and ROI has been selected by CAD and confirmed by the physician, then the ROI has a higher probability than an ROI selected by the physician without the aid of CAD. In this case, according to some embodiments, the CAD and physician selected ROI is displayed with an increased emphasis. For example, shape, size, color, and/or numerical probability could be used for such increased emphasis. When the CAD is available to both modalities, mammogram and 3D ultrasound, and when an ROI has bee selected by CAD from both modalities, then such an ROI has an even higher probability, and can be displayed with even greater emphasis.

Display600corresponds to the display screen151of device150shown inFIG. 1D. As shown inFIG. 6, display600includes ultrasound images610,620and630as well as user positionable cursors612,622and632. In response to the user's positioning of one of the cursors612,622or632, the corresponding location on the mammogram MLO view640is automatically calculated and an ROI marker642is displayed. Note the MLO view640also includes a nipple marker644. According to some embodiments, a CC mammography view can be used instead of, or in addition to MLO view640, with an ROI marker displayed thereon. For example, when displaying both CC and MLO views the two images can be arranged one above the other or side by side, depending on the aspect ratio of the display screen device and based on the user's preference. According to some embodiments, an integrated viewing system such as shown inFIG. 1Dcan be used to automatically calculate and display locations on ultrasound images that correspond to a user selected ROI on mammogram images. In this case, the user selects the location of the ROI on two or more displayed mammogram images. The system automatically calculates the coordinates in the 3D ultrasound image that corresponds to the selected ROI. The system then automatically selects the appropriate coronal view and other 2D orthogonal images and displays them to the user. The system also automatically displays ROI markers (such as cross hairs or dotted circles) on the displayed 2D ultrasound images that correspond to the calculated ROI location. It has been found that automatic cross modality location identification greatly increase the user's efficiency, while at the same time decreasing the likelihood of errors made by the user.

FIG. 7is a flow chart illustrating aspects of creating and displaying visual aids for indicating locations on ultrasound images that correspond to selected ROIs on mammogram images, when separate viewing devices are used, according to some embodiments. In step710, the mammogram and ultrasound images of the same patient are loaded on the mammogram viewing device. In step712, the nipple locations are marked on all the views of mammograms (CC, MLO, etc.) and on the coronal view of the ultrasound. According to some embodiments this done automatically by nipple detection software. According to other embodiments, this is done manually by the radiologist or by some other user. In step714, the breast icons are automatically created for all mammograms based on the mammogram images. In step716the breast icons are automatically created based on the ultrasound images. In step718the lesion or ROI is identified on at least two views, CC and MLO for example. According to some embodiments, the ROIs are both selected by the user. According to some other embodiments either the location of the ROI in one or more of the mammogram views is pre-identified, suggested and/or selected with the aid of CAD software. In step720, the 3D coordinates of the ROI on 3D ultrasound image is automatically calculated based on ROI coordinates on the mammograms, nipple coordinates on mammograms, angles of imaging, thickness of the breast and other imaging geometric information. A more detailed description of algorithms for calculating the coordinates is described in co-pending U.S. patent application Ser. No. 12/839,371. In step722the location (using for example, clock position and distance from the nipple) of the corresponding ROI is indicated to the user on the coronal view icon (clock face) based on the calculated 3D coordinates.

FIG. 8is a flow chart illustrating aspect of creating and displaying visual aids for indicating locations on mammogram images that correspond to selected ROIs on ultrasound images when separate viewing devices are used, according to some embodiments. In step810, the ultrasound and mammograms of the same patient are loaded on the ultrasound viewing device. In step812, the nipple location is marked on the ultrasound and mammogram images. As described, supra, this can be done automatically by nipple detection software or manually by the user. In steps814and816the mammogram icons are automatically created based on the mammogram images and the ultrasound icons are automatically created based on the ultrasound images. In step818the ROI is identified on one of the ultrasound views, for example, the coronal view. This will be used determine the 3D coordinates of the lesion. According to some embodiments, the ROIs are both selected by the user. According to some embodiments, the ROI is selected by the user. According to some other embodiments the location of the ROI in the ultrasound image view(s) is pre-identified, suggested and/or selected with the aid of CAD software. In step820the coordinates of the ROI including distance from the nipple and angle from a predetermined axis is automatically calculated on all mammography views i.e. CC, MLO etc., based on ROI coordinates, nipple coordinates on ultrasound, and angles of imaging, thickness of the breast, nipple location and other imaging geometric information of the mammograms. A more detailed description of algorithms for calculating the coordinates is described in co-pending U.S. patent application Ser. No. 12/839,371. In step822, the location of the corresponding ROI on the mammogram images or on the mammogram icons is indicated based on the calculated coordinates.

FIG. 9is a flow chart illustrating aspect of creating and displaying visual aids indicating locations on images of one modality that correspond to selected ROIs on another modality when an integrated viewing device is used, according to some embodiments. In step910, the ultrasound and mammograms of the same patient are loaded onto the integrated viewing device. In step912, the nipple location on the ultrasound and mammogram images. As described, supra, this can be done automatically by nipple detection software or manually by the user. In cases where correlation is being calculated from ultrasound images to mammogram images, in step916the ROI on one of the ultrasound views, for example, the coronal view is identified. This will determine the 3D coordinate of the lesion. According to some embodiments, the ROI is selected by the user. According to some other embodiments the location of the ROI in the ultrasound image view(s) is pre-identified, suggested and/or selected with the aid of CAD software. In step918the coordinates of the ROI including distance from the nipple and angle from a predetermined axis on all mammography views i.e. CC, MLO etc., is automatically calculated based on ROI coordinates, nipple coordinates on ultrasound, and angles of imaging, thickness of the breast, nipple location and other imaging geometric information of the mammograms. A more detailed description of algorithms for calculating the coordinates is described in co-pending U.S. patent application Ser. No. 12/839,371. In step920, the location of the corresponding ROI on the mammogram images is indicated based on the calculated coordinates calculated in step918.

In cases where correlation is being calculated from mammogram images to ultrasound images, in step922, the lesion or ROI on at least two views, CC and MLO for example, is identified. According to some embodiments, the ROIs are both selected by the user. According to some other embodiments either the location of the ROI in one or more of the mammogram views is pre-identified, suggested and/or selected with the aid of CAD software. In step924the 3D coordinates of the ROI on the 3D ultrasound image is automatically calculated based on ROI coordinates, nipple coordinates on mammograms, angles of imaging, thickness of the breast and other imaging geometric information. A more detailed description of algorithms for calculating the coordinates is described in co-pending U.S. patent application Ser. No. 12/839,371. In step926, the location of the corresponding ROI on the 3D breast ultrasound is indicated directly on the 2D ultrasound images based on the calculated 3D coordinates from step924. As describe, supra, the system is configured to automatically select the appropriate coronal view and other 2D orthogonal images and displays them to the user. The system also automatically displays ROI markers (such as cross hairs or dotted circles) on the displayed 2D ultrasound images that correspond to the calculated ROI location.

According to some embodiments, the ROI locations between breast tomosynthesis and 3D breast ultrasound is automatically calculated and displayed to the user. The method is similar to the case of correlation between mammography and ultrasound as described herein above. The difference is that since tomosynthesis is a 3D modality, one needs only to identify the ROI location in one of the tomo views as opposed to two 2D mammography views.

Various modifications may be made without departing from the spirit and scope of the invention. Accordingly, the invention is not limited to the above-described embodiments, but instead is defined by the appended claims in light of their full scope of equivalents.