Patent ID: 12249417

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will now be described by reference to the drawings.

(1) Outline of Embodiment

An ultrasonic diagnostic device according to an embodiment includes a detection unit, a database, a determination unit, and a display control unit. The detection unit has a machine learned detection model, and detects a candidate for a lesion portion, or a lesion candidate, in an ultrasonic image. The database contains features of lesion candidates to be excluded; that is, lesion candidates that are exclusion targets, registered therein for every lesion candidate to be excluded. The determination unit collates a feature of a detected lesion candidate with the features registered in the database, to thereby determine whether the detected lesion candidate is an exclusion target. If the detected lesion candidate is not an exclusion target, the display control unit displays a mark for notifying the examiner of, or indicating, the detected lesion candidate on the ultrasonic image. Meanwhile, if the detected lesion candidate is an exclusion target, the display control unit restricts display of the mark. Note that the detection unit corresponds to a detector; the determination unit corresponds to a determiner; and the display control unit corresponds to a controller.

The above-described structure excludes some of the detection targets from targets to be marked, or marking targets, through post hoc processing so that the detection targets are not determined as marking targets. In other words, the above-described structure can readily restrict marking targets without modification of the machine learned detection model. In addition, this structure can readily release the restriction.

Note that the concept of a feature of a lesion candidate includes a feature to be extracted from a lesion candidate, that to be extracted from an image portion containing a lesion candidate and an area around the lesion candidate, and that to be extracted from an ultrasonic image containing a lesion candidate. Also, the concept of a feature includes a feature vector, an image pattern, and so forth. A mark for indicating a lesion candidate is a display element for specifying a lesion candidate in an ultrasonic image.

An ultrasonic diagnostic device according to this embodiment further includes an input unit and a registration unit. The input unit receives an instruction for exclusion, or an exclusion instruction, sent from a user who is observing a lesion candidate in an ultrasonic image and sees a mark on the ultrasonic image. The registration unit registers a feature of a lesion candidate addressed by an exclusion instruction in the database. This structure enables correction by a user, of lesion candidates to be marked; that is, lesion candidates that are marking targets, without relearning of a machine learned detection model. Note that the input unit corresponds to an inputter, and the registration unit corresponds to a register.

In this embodiment, the input unit receives an exclusion instruction in a frozen state. The registration unit registers a feature of a lesion candidate addressed by the exclusion instruction in the database in the frozen state to thereby update the database. Note that the frozen state is a state in which transmission and reception is not carried out or in which a static image is kept displayed.

In this embodiment, the input unit receives an exclusion instruction in a real time operation state. In the real time operation state, the registration unit specifies one or more sets of displayed frame data, based on the time of reception of the exclusion instruction. In addition, the registration unit in the real time operation state extracts a feature of a lesion candidate from the one or more sets of specified frame data, and registers the extracted feature in the database to thereby update the database.

The above-described structure can update the database in the real time operation state. For example, this structure can restrict display of a mark when a lesion candidate is erroneously detected. A real time operation state is a state in which an ultrasonic image is displayed as a motion image and transmission and reception are being carried out. For example, a plurality of sets of frame data over a predetermined period of time in the past immediately preceding the time of reception of an exclusion designation as a referential point are specified, and a plurality of features or a representative feature are extracted.

In this embodiment, the detection unit has a function of obtaining a feature of a lesion candidate. Alternatively, a calculation unit may be provided separately from the detection unit to calculate the feature of a lesion candidate. Note that the calculation unit corresponds to a calculator.

The ultrasonic diagnostic device according to this embodiment includes a management unit for clearing the entirety or a part of the database. In this embodiment, the database includes a plurality of tables, in each of which at least one feature is registered. The management unit selects a table to clear from among the plurality of tables. For example, a table may be provided for every examinee, for every ultrasonic examination, or for every examiner, and a table having been used may be cleared at a predetermined time. The management unit corresponds to a manager.

A diagnostic assisting method according to this embodiment includes a detection step, a registration step, a determination step, and a display control step. In the detection step, a lesion candidate in an ultrasonic image is detected using a machine learned detection model. In the registration step, features of lesion candidates to be excluded is registered in the database. In the determination step, a feature of the detected lesion candidate is collated with the features registered in the database to thereby determine whether the detected lesion candidate is to be excluded, or is an exclusion target. In the display control step, a mark for indicating the detected lesion candidate is displayed on the ultrasonic image when the detected lesion candidate is not an exclusion target. Meanwhile, when the detected lesion candidate is an exclusion target, display of the mark is restricted.

The above-described diagnostic assisting method is implemented as a function of hardware or software. In the latter case, a program for executing the diagnostic assisting method is installed into an information processing device via a portable storage medium or a network. The concept of an information processing device includes ultrasonic diagnostic devices, image processing devices, computers, and so forth. The information processing device includes a non-temporary storage medium storing the above-described program.

(2) Details on Embodiment

FIG.1is a block diagram illustrating the structure of an ultrasonic diagnostic device according to this embodiment. The ultrasonic diagnostic device is a medical device to be installed in medical institutes, such as hospitals, and forms ultrasonic images based on a reception signal obtained from transmitted and received ultrasonic waves with respect to a living body (an examinee). Target organs of an ultrasonic examination include breasts, for example.

In a group medical checkup for breasts, it is necessary to specify lesion portions in a short period of time without failure. The ultrasonic diagnostic device according to this embodiment is equipped with Computer Aided Detection (CADe) for automatic detection of a lesion candidate (for example, a portion with low brightness that may be a tumor) contained in an ultrasonic image to assist specification of a lesion portion by an examiner, to be described later in detail.

A probe10functions as a means for transmitting and receiving ultrasonic waves. Specifically, the probe10is a portable transceiver, and is to be held and operated by a user, or an examiner (doctors, examination technicians, and so forth). In ultrasonic diagnosis of breasts, the probe10is placed such that its transducing surface (specifically, the surface of an acoustic lens) abuts on the surface of the chest of an examinee, so that tomographic images are displayed on a real time basis. While observing the tomographic images displayed on a real time basis, the user manually moves the probe10for scanning on the surface of the chest. When a lesion candidate is specified in a tomographic image, the position and posture of the probe10are slowly adjusted so that the tomographic image is carefully observed with the probe10at a fixed position and posture.

The probe10of the illustrated exemplary structure includes a transducer array composed of a plurality of one-dimensionally aligned transducers. The transducer array produces an ultrasonic beam (a transmission beam and a reception beam)12. Electrical scanning with an ultrasonic beam12forms a scanning surface14. The scanning surface14is a surface for observation; that is, a two dimensional (2D) data capturing area. Known examples of an electronic scanning method using an ultrasonic beam12include an electronic sector scanning method, an electronic linear scanning method, and so forth. Convex scanning with an ultrasonic beam12may be applied. The probe10may incorporate a 2D transducer array so that volume data from the inside of a living body can be obtained through 2D scanning with an ultrasonic beam.

A positioning system for obtaining the position information of the probe10may be provided. A positioning system is composed of, for example, a magnetic sensor and a magnetic field generator. In this case, the probe10(precisely speaking, the probe head of the probe10) has a magnetic sensor. The magnetic sensor detects a magnetic field formed by the magnetic field generator to provide three-dimensional (3D) coordinate information. Based on the 3D coordinate information, the position and posture of the probe10can be specified.

A transmission circuit22functions as a transmission beam former. Specifically, in transmission, the transmission circuit22supplies a plurality of transmission signals in a parallel manner to the transducer array, which produces a transmission beam. Meanwhile, in reception, when the transmission waves is reflected on the inside of a living body and returns to reach the transducer array, the plurality of transducers output a plurality of reception signals in parallel. A receiving circuit24functions as a reception beam former, and generates beam data through phase-alignment and summing (also known as delay and summing) with a plurality of reception signals.

For every electronic scanning, a plurality of sets of beam data aligned in the direction of the electronic scanning are formed, which constitute received frame data corresponding to the scanning surface14. Each set of beam data is composed of a plurality of sets of echo data aligned in the depth direction. Downstream the receiving circuit24, a beam data processing unit, not illustrated, is provided.

An image forming unit26is an electronic circuit for forming a tomographic image (a B-mode tomographic image), based on the received frame data. The image forming unit26has a digital scan converter (DSC). The DSC has a coordinate conversion function, a pixel interpolation function, a frame rate conversion function, and so forth. In further detail, the image forming unit26forms a display frame data array, based on the received frame data array. Each set of display frame data constituting the display frame data array is tomographic image data. A plurality of sets of tomographic image data constitute a real time motion image. Note that ultrasonic images other than tomographic images may be formed. For example, a color flow mapping image may be formed. A 3D image representing tissues in a 3D manner may be formed. In the illustrated exemplary structure, the display frame data array is sent to a display processing unit32and an image analyzing unit28.

The image analyzing unit28is a module for implementing a CADe function. The image analyzing unit28has a detection unit33and an exclusion processing unit34.

The detection unit33detects a lesion candidate for every set of display frame data; that is, for every tomographic image. The detection unit33includes a machine learning detection unit having a machine learned detection model. The detection unit33is composed of, for example, a convolutional neural network (CNN) or the like, and detects a closed area with low brightness as a candidate for a lesion portion (hereinafter referred to as a lesion candidate). Prior to detection of a lesion candidate by the detection unit33, binary processing, noise removal, or the like may be applied to a tomographic image. The detection unit33outputs lesion candidate information after detection of a lesion candidate.

Lesion candidate information includes the position information of a lesion candidate, the size information of a lesion candidate, and credibility information on a detection, or the like. When credibility of a level higher than a predetermined level is obtained, detection of a lesion candidate is determined. The detection unit33has a function of calculating a feature vector as the feature of a lesion candidate.

The position information of a lesion candidate is, for example, coordinate information indicating the central point of the lesion candidate itself. Alternatively, the position information is coordinate information indicating the central point of a figure encircling a lesion candidate so as to be in contact with the lesion candidate. Note that the central point is a representative point. Alternatively, the central point can be a geometrical central point or the barycentric point of the figure. The size information of a lesion candidate is, for example, information indicating the size of the lesion candidate itself or that of a figure encircling the lesion candidate while being in contact with the lesion candidate. For example, the size of a lesion candidate can be specified based on the coordinates of the central point of the figure and those of the upper left corner of the figure. Providing that the coordinates of the central point are specified, the coordinates of the upper left corner of the figure may be considered the size information of the lesion candidate. Alternatively, the area of a lesion candidate may be obtained as the size information of the lesion candidate. A plurality of lesion candidates may be detected in parallel.

A feature vector is composed of a plurality of vector elements. For calculation of feature vectors, one or more publicly known feature vector calculation methods can be employed. For example, a multivariate analysis may be employed as a feature vector calculation method. An image pattern may be used as a feature. Examples of the feature of a lesion candidate include a feature to be extracted from the lesion candidate, that to be extracted from an image portion containing the lesion candidate and an area around the lesion candidate, and that to be extracted from an ultrasonic image containing the lesion candidate.

The exclusion processing unit34has an exclusion database. The exclusion processing unit34collates the feature vector of a lesion candidate presently detected by the detection unit33(a present lesion candidate) with the feature vectors contained in the exclusion database. In the case where a feature vector having a similarity of a level higher than a predetermined level is found; in other words, in the case where the detected lesion candidate is an exclusion target, the exclusion processing unit34outputs an exclusion instruction signal to a mark display control unit30. Meanwhile, in the case where a feature vector having a similarity of a level higher than the predetermined level is not found; in other words, in the case where the detected lesion candidate is not an exclusion target, no exclusion instruction signal is outputted.

The mark display control unit30displays, or superimposes, a mark for indicating a detected lesion candidate on an ultrasonic image. The mark display control unit30, however, does not display a mark in the case where the exclusion processing unit34outputs an exclusion instruction signal. As described above, a marking target is discriminated from a detection target, which enables post hoc customization for marking targets. In customization, relearning by the machine learned detection unit in the detection unit33is unnecessary. Graphic data containing a mark generated by the mark display control unit30are outputted to the display processing unit32.

In displaying marks, the marks may be displayed in different manners to thereby express difference in degree of credibility of the detection. For example, for low credibility, the mark may be displayed in a cold color, and for high credibility, in a warm color. Alternatively, for low credibility, the mark may be displayed with lower brightness, and for high credibility, with higher brightness. Alternatively, for low credibility, the mark may be displayed with high transparency, and for high credibility, with low transparency. Alternatively, for low credibility, the mark may be displayed in a thin line, and for high credibility, in a thick line. Alternatively, for low credibility, the mark may be displayed in a broken line, and for high credibility, in a solid line. Alternatively, marks of different types may be switched with one another for displaying. For example, display of a rectangular figure may be switched with display of four display elements indicative of four respective corners of the rectangular figure. Two or more manners for displaying may be applied simultaneously.

The image forming unit26, the image analyzing unit28, and the mark display control unit30each include a processor. Alternatively, a single processor may function as the image forming unit26, the image analyzing unit28, and the mark display control unit30. A central processing unit (CPU) to be described later may function as the image forming unit26, the image analyzing unit28, and the mark display control unit30.

The display processing unit32has a color operation function, an image combining function, and so forth. The display processing unit32is supplied with an output from the image forming unit26and an output from the mark display control unit30. Note that a mark encircling a lesion candidate is one element constituting a graphic image. Although in the present embodiment the mark display control unit30generates a mark, a main control unit38, the display processing unit32, or the like may generate a mark.

A display36includes a liquid crystal display (LCD), an organic electro-luminescent (EL) display, or the like. The display36shows a tomographic image as a motion image on a real time basis, and also a mark as a part of a graphic image. The display processing unit32includes a processor, for example.

The main control unit38controls the operations of the respective structural components illustrated inFIG.1. The main control unit38includes a central processing unit (CPU) for executing a program in this embodiment. The main control unit38is connected to an operation panel40. The operation panel40is an input unit; that is, an input device, and has a plurality of switches, a plurality of buttons, a trackball, a keyboard, or the like. A condition for displaying a mark, or a mark display condition, can be set or changed with the operation panel40. The operation panel40is operated to register an exclusion target in the exclusion database.

Although in the present embodiment a display frame data array is inputted to the image analyzing unit28, a received frame data array may be inputted to the image analyzing unit28(refer to reference numeral42). In this case, another image forming unit may be provided separately from the image forming unit26to readily and promptly form images.

A cine-memory27has a ring buffer structure. The cine-memory27temporarily stores display frame data arrays within a predetermined period of time in the past immediately preceding the present time. In a frozen state to be described later, display frame data selectively read from the cine-memory27is displayed on the display36as a tomographic image (a static image). During display, the image analyzing unit28and the mark display control unit30may be operated again to register an exclusion target. Data generated in the image analyzing unit28and the mark display control unit30may be temporarily stored in a buffer, so that registration of an exclusion target can be performed using data read from the buffer.

FIG.2illustrates a mark generation method. A tomographic image44contains a lesion candidate46. The tomographic image44is binarized to generate a binary image. Edge detection or area detection is applied to the binary image to extract a binarized lesion candidate46A. For example, the coordinates of both ends of the lesion candidate46A in the horizontal direction and those in the vertical direction together define a rectangle52that is circumscribed about the lesion candidate46A. In actuality, the coordinates of the central point48of the rectangle52and those of the upper left corner50of the rectangle52are specified.

Further, another rectangle54is defined outside the rectangle52with predetermined margins56,58in the horizontal and vertical directions, respectively. Specifically, the rectangle54is displayed as a mark64on the tomographic image44. The mark64is a figure encircling the lesion candidate46and an area around the lesion candidate46. In the illustrated example, the mark64is drawn in a broken line. Alternatively, a mark composed of four elements representing four respective corners of a rectangle may be displayed. Alternatively, a round or oval mark may be displayed.

In this embodiment, detection of a lesion candidate46is conducted in units of display frame data. In the case where a lesion candidate46is detected and the detected lesion candidate46is not an exclusion target, a mark64is displayed on the tomographic image44containing the lesion candidate46. Displaying the mark64enables an examiner to be aware of the presence of a lesion candidate46. This can prevent overlooking of a lesion candidate46. Meanwhile, in the case where the detected lesion candidate46is an exclusion target, no mark64is displayed.

FIG.3illustrates a first exemplary structure of the image analyzing unit28illustrated inFIG.1. The detection unit33has a machine learned detection model70. Using the detection model70, a lesion candidate contained in a tomographic image (display frame data)72is detected. The detection model70may contain a CNN parameter set, for example. Note that installation of a CNN parameter set is indicated with reference numeral74. In this embodiment, the exclusion processing unit34is provided to enable correction of a marking target without relearning of the machine learned detection model70. Position information, size information, and so forth, which constitute the lesion portion information, are inputted to the mark display control unit30, as indicated with reference numeral76.

The detection unit33has a function of calculating the feature vector of a lesion candidate. Information indicating a feature vector is sent to the exclusion processing unit34, as indicated with reference numeral78. Alternatively, a module different from the detection unit33may be provided to calculate a feature vector, to be described later with reference toFIG.8.

The exclusion processing unit34includes an exclusion database80, a collating unit (a determination unit)82, and a registration unit84. A management unit89, to be described later, may constitute a part of the exclusion processing unit34. In the illustrated exemplary structure, the exclusion database80is composed of a plurality of exclusion tables80A. In actuality, an exclusion table80A selected in response to a selection signal88is used. Needless to say, the exclusion database80may be composed of a single exclusion table. A plurality of exclusion tables80A corresponding to a plurality of respective examinees may be provided; a plurality of exclusion tables80A corresponding to a plurality of respective examiners may be provided; or a plurality of exclusion tables80A corresponding to a plurality of respective medical specialties may be provided.

The collating unit82collates the feature vector of the presently detected lesion candidate with the feature vectors registered in the selected exclusion table80A. Specifically, in the case where collation finds a feature vector of the present lesion candidate having a similarity of a level higher than a predetermined level, the present lesion candidate is determined as an exclusion target, and the collating unit82outputs an exclusion instruction signal90to the mark display control unit30. Meanwhile, in the case that there is no feature vector having a similarity of a level higher than the predetermined level, no exclusion instruction signal90is outputted. As described above, the present lesion candidate can be evaluated post hoc in this embodiment. In other words, such a post hoc evaluation can eliminate the need for relearning of the detection model70.

The registration unit84registers the feature vector of the present lesion candidate in the selected exclusion table. Registration of a feature vector can be performed in either the frozen state or the real time operation state, to be described later. Reference numeral86indicates a signal indicating a registration instruction issued by a user.

The management unit89clears the entirety or a part of the exclusion database80. Specifically, the content of the exclusion table80A can be cleared for every exclusion table80A. For example, one or more exclusion tables80A may be cleared in response to a clear instruction sent from a user. Alternatively, one or more exclusion tables80A may be automatically cleared once a predetermined clearing condition is satisfied. For example, automatic clearing may be applied, for example, upon start of examination of a new examinee, upon change in a transmission/reception condition, or the like.

The mark display control unit30has a mark generation function and a mark display restriction function (generally, a function of not displaying a mark). The mark display control unit30generates a mark in the form of a figure encircling a lesion candidate, based on the lesion candidate information outputted from the detection unit33. A mark may be displayed in a different manner, depending on credibility. Upon receipt of an exclusion instruction signal, the mark display control unit30does not generate a mark and does not display a mark. A mark is generated in units of a frame, and is determined to be displayed or not in units of a frame.

FIG.4schematically illustrates a first exemplary operation to be executed in a frozen state. A tomographic image102is displayed as a static image on a screen92. That is, a freeze button96in the operation panel40has already been operated and a frozen state, in which transmission/reception is suspended, as described above, has been thereby set. In addition, a band figure is shown below the tomographic image102, representing the storage area of the cine-memory. The right end of the band corresponds to the time at which a frozen state is set. Shifting a cursor100farther leftward along the band enables selection of display frame data in a farther past. Selected display frame data are displayed as the tomographic image102. In the illustrated example, the tomographic image102contains a lesion candidate104, with a mark106displayed encircling the lesion candidate104.

If the mark106is displayed due to erroneous detection of a lesion candidate or if marking of the detected lesion candidate is not desired, an exclusion button108is operated. Then, the feature vector of the displayed lesion candidate is registered in the exclusion database (specifically, in the selected exclusion table), whereby the exclusion database is updated. Registration of exclusion targets is repetitively applied in the frozen state, when necessary. Re-operating the freeze button96releases the frozen state, with the real time operation state restored. In the real time operation state, whether a lesion candidate is an exclusion target is determined, based on the updated database.

FIG.5is a flowchart of the first exemplary operation. In S10, a determination is made as to whether the freeze button has been operated, upon which the ultrasonic diagnostic device is put in the frozen state. In S12, a user performs a back-search with respect to the cine-memory to select specific display frame data. In S14, a determination is made as to whether an operation for instructing registration has been executed. Upon determination that an operation for registration has been executed, in S16the feature vector of the displayed lesion candidate is registered in the exclusion database. That is, the exclusion database is updated. Thereafter, a series of processes in S10and thereafter is repetitively executed until a determination is made in S18that an operation for releasing the frozen state has been executed. After determination in S18that an operation for releasing the frozen state has been executed, determination for a lesion candidate will be thereafter executed based on the updated exclusion database, as indicated with reference numeral110.

FIG.6schematically illustrates a second exemplary operation to be executed in the real time operation state, in which (A) corresponds to a timeline, with t1indicating a time (the present time) when an exclusion operation is executed, and (B) corresponds to a cine-memory, where a display data array112is stored. Reference numeral112A indicates the leading display frame data; that is, the latest display frame data stored in the cine-memory. In the second exemplary operation, an exclusion operation initiates registration of exclusion targets over a predetermined period of time ta in the past immediately preceding time t1, as indicated with (C). Note that the predetermined period of time ta may be, for example, N seconds, wherein N is one, two, three, and so forth, for example.

A determination is made as to whether a lesion candidate has been detected in each of the sets of display frame data in the predetermined period of time ta. Upon determination that a lesion candidate has been detected, the feature vector of the detected lesion candidate is automatically registered in the exclusion database. Specifically, a single feature vector may be calculated based on a plurality of lesion portions contained in a plurality of sets of respective display frame data, and registered in the database, the single feature vector representing the plurality of lesion portions. For example, in the case where a seemingly erroneous detection is performed in an ultrasonic examination, an examiner applies an exclusion operation (for example, by pressing an exclusion button). Then, a feature vector is registered in the database even in the real time operation state, whereby the database is updated. Thereafter, mark display control is executed based on the updated database.

FIG.7is a flowchart of the second exemplary operation. In S20, a determination is made as to whether an exclusion operation has been performed. Upon determination that an exclusion operation has been performed, a reference period is decided in S22, and respective sets of display frame data within the decided reference period are checked in S24. In the case where any lesion candidate is contained in the display frame data, the feature vector of the contained lesion candidate is registered in the exclusion database, whereby the exclusion database is sequentially updated. That is, the updated exclusion database is instantly validated for use, as being indicated with reference numeral114. In S28, a determination is made is as to whether to end the registration processing.

FIG.8illustrates a second exemplary structure of an image analyzing unit. InFIG.8, structural components similar to those illustrated inFIG.3are assigned the same or corresponding reference numerals, and are not described again.

An image analyzing unit28A includes a detection unit33A and an exclusion processing unit34A, and further a feature vector calculation unit116. The feature vector calculation unit116is provided separately from the detection unit33A, and calculates a feature vector of a lesion candidate. The feature vector calculation unit116is not a machine learning calculation unit, but a general calculation unit. The detection unit33A has a machine learned detection model70A, but does not have a function of calculating a feature vector. In the second exemplary structure as well, the exclusion processing unit34A executes an exclusion processing, using the exclusion database80. A management unit is not illustrated inFIG.8.

FIG.9illustrates a modified example of the display of the image. In the real time operation state, a tomographic image120as a motion image is displayed. The displayed tomographic image120contains a lesion candidate, or a detection target, with a mark126not displayed. In this case, an indicator128for indicating that the exclusion processing mode is valid may be displayed. Alternatively, an indicator130for indicating that displaying a mark is restricted may be displayed.

According to the above-described embodiment, a lesion candidate to be marked can be flexibly customized without relearning by a machine learning detection unit. An additional database may be provided together with, or in place of, the exclusion database.