Patent ID: 12191036

BEST MODE

A method of displaying an ultrasonic image according to disclosed embodiments includes identifying a lesion area included in the ultrasonic image; diagnosing the lesion area to acquire a diagnostic result; displaying a first area in the lesion area, which is at least one area that is the basis of diagnosing a lesion, in the lesion area of the ultrasonic image and generating a diagnostic image; and displaying a user interface screen including the diagnostic image and the diagnostic result.

In addition, when a plurality of features indicating a feature of the lesion in the lesion area are present, the first area may be an area indicating at least one among the plurality of features.

In addition, the first area may be an area that is the basis of determining at least one feature among a shape of the lesion, an orientation of the lesion, a margin of the lesion, an echo with respect to the lesion area, and a posterior of the lesion area.

In addition, the diagnostic result may include information indicating whether the lesion is benign or malignant.

In addition, the generating of the diagnostic image may include, when a plurality of features indicating the lesion area exist, displaying at least one area in the lesion area using at least one among different colors, marks, symbols, transparency, echoes, types of dotted line, and thicknesses of the dotted lines, thereby distinguishing different features from each other among the plurality of features.

In addition, the plurality of features may include at least one among a shape of the lesion, an orientation of the lesion, a margin form of the lesion, an echo with respect to the lesion area, and a posterior feature of the lesion area that are the basis of diagnosing the lesion.

In addition, the generating of the diagnostic image may include, when the feature of the lesion area appearing in a second area included in the lesion area is changed and thus the diagnostic result is changed, displaying the second area in the ultrasonic image and generating the diagnostic image.

In addition, the generating of the diagnostic image may further include displaying the first area to be distinguished from the second area and generating the diagnostic image.

In addition, the displaying of the user interface screen may include displaying the user interface screen including a list including a plurality of items corresponding to a plurality of features indicating the diagnostic image, the diagnostic result, and the lesion.

In addition, the generating of the diagnostic image may further include, in response to a user input for selecting any one among the plurality of items, displaying at least one partial area indicating a feature corresponding to the selected item as the first area in the ultrasonic image and generating the diagnostic image.

In addition, in the list, the plurality of items may include information specifically indicating features corresponding to the plurality of items, and each of the plurality of items may be displayed to match at least one partial area corresponding thereto.

In addition, the displaying of the user interface screen may further include, when a first feature included in the list is changed and thus the diagnostic result is changed, displaying the first feature included in the list to be distinguished from other features.

In addition, the generating of the diagnostic image may include displaying diagnostic information, which is information on the feature of the lesion area appearing in the first area, in the ultrasonic image and generating the diagnostic image.

In addition, acquiring the diagnostic information may include analyzing the ultrasonic image using at least one of a computer aided detection & diagnosis (CAD) technology and an artificial intelligence (AI) technology and acquiring at least one among the lesion area, the diagnostic result, and diagnostic information corresponding to the diagnostic result.

In addition, the identifying of the lesion area may include identifying a contour of the lesion area, and the method of displaying an ultrasonic image may further include, when a plurality of contours exist as the identification result, displaying the plurality of contours in the ultrasonic image and generating a plurality of sub-images; and outputting a user interface screen including the plurality of sub-images.

An ultrasound diagnosis device may include a display; a memory configured to store at least one instruction; and a processor configured to execute at least one of the at least one instruction to identify a lesion area included in an ultrasonic image, diagnose the lesion area to acquire a diagnostic result, display a first area, which is at least one area that is the basis of diagnosing a lesion in the lesion area, in the lesion area of the ultrasonic image to generate a diagnostic image, and control a user interface screen including the diagnostic image and the diagnostic result to be displayed through the display.

A computer program product may include a computer program including commands executable by a computer and configured to be stored in a recording medium to execute a method of displaying an ultrasonic image on a computer, the computer program may include identifying a lesion area included in an ultrasonic image; diagnosing the lesion area to acquire a diagnostic result; displaying a first area in the lesion area, which is at least one area that is the basis of diagnosing a lesion, in the lesion area of the ultrasonic image and generating a diagnostic image; and displaying a user interface screen including the diagnostic image and the diagnostic result.

Modes of the Invention

The present specification describes the principles of the present disclosure and discloses embodiments such that the scope of the present disclosure may be clarified and those skilled in the art to which the present disclosure pertains may implement the present disclosure. The disclosed embodiments may be implemented in various forms.

Like reference numerals refer to like components throughout the present specification. The present specification does not describe all components of embodiments, and a common description in the technical field to which the present disclosure pertains and an overlapping description between the embodiments will be omitted. The term “part” or “portion” used herein may be implemented in software or hardware, and according to embodiments, a plurality of “parts” or “portions” may be implemented as one unit or one element, or one “part” or “portion” may include a plurality of units or elements. Hereinafter, an operation principle and embodiments of the present disclosure will be described with reference to the accompanying drawings.

In the present specification, an “image” may include a medical image obtained by a medical imaging device such as a magnetic resonance imaging (MRI) device, a computed tomography (CT) device, an ultrasound imaging device, and an X-ray imaging device.

In the present specification, an “object” is to be photographed and may include a person, an animal, or a part thereof. For example, the object may include a part (organ) of a human body, a phantom, or the like.

Throughout the specification, an “ultrasonic image” means an image of the object, which is processed based on an ultrasonic signal transmitted to the object and reflected from the object.

Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

FIG.1is a block diagram illustrating a configuration of an ultrasound diagnosis device according to an embodiment.

FIG.1is a block diagram illustrating a configuration of an ultrasound diagnosis device100according to an embodiment. The ultrasound diagnosis device100may include a probe20, an ultrasonic transceiver110, a controller120, an image processing unit130, a display unit140, a storage unit150, a communication unit160, and an input unit170.

The ultrasound diagnosis device100may be implemented as a portable type as well as a cart type. Examples of a portable ultrasound diagnosis device may include a smart phone, a laptop computer, a personal digital assistant (PDA), a tablet personal computer (PC), and the like including a probe and an application, but the present disclosure is not limited thereto.

The probe20may include a plurality of transducers. The plurality of transducers may transmit ultrasonic signals to an object10according to a transmission signal applied from a transmission unit113. The plurality of transducers may receive ultrasonic signals reflected from the object10to form a reception signal. Further, the probe20may be implemented integrally with the ultrasound diagnosis device100or may be implemented as a separate type in which the probe20is connected to the ultrasound diagnosis device100in a wired or wireless manner. Further, the ultrasound diagnosis device100may include one or more probes20according to an implementation form.

The controller120controls the transmission unit113to form a transmission signal to be applied to each of the plurality of transducers in consideration of the positions and focal points of the plurality of transducers included in the probe20.

The controller120controls a reception unit115to convert a reception signal received from the probe20in an analog-to-digital conversion manner and to sum the digitally converted reception signal in consideration of the positions and focal points of the plurality of transducers, thereby generating ultrasonic data.

The image processing unit130generates an ultrasonic image using the ultrasonic data generated by the ultrasonic reception unit115.

The display unit140may display the generated ultrasonic image and various pieces of information processed by the ultrasound diagnosis device100. The ultrasound diagnosis device100may include one or more display units140according to an implementation form. Further, the display unit140may be implemented as a touch screen in combination with a touch panel.

The controller120may control the overall operation of the ultrasound diagnosis device100and a signal flow between internal components of the ultrasound diagnosis device100. The controller120may include a memory that stores a program or data for performing a function of the ultrasound diagnosis device100and a processor that processes the program or data. Further, the controller120may control the operation of the ultrasonic diagnosis device100by receiving a control signal from the input unit170or an external device.

The ultrasound diagnosis device100may include the communication unit160and may be connected, through the communication unit160, to an external device (for example, a server, a medical device, a portable device (a smart phone, a tablet PC, a wearable device, and the like)).

The communication unit160may include one or more components enabling communication with the external device and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

The communication unit160may receive a control signal and data from the external device and transmit the received control signal to the controller120so that the controller120may control the ultrasound diagnosis device100in response to the received control signal.

Alternatively, the controller120may transmit a control signal to the external device through the communication unit160so that the external device may be controlled in response to the control signal of the controller120.

For example, the external device may process the data of the external device in response to the control signal of the controller received through the communication unit.

A program capable of controlling the ultrasound diagnosis device100may be installed in the external device, and the program may include instructions for performing some or all of the operations of the controller120.

The program may be installed in the external device in advance or may be installed by a user of the external device by downloading the program from a server that provides applications. The server that provides applications may include a recording medium in which the corresponding program is stored.

The storage unit150may store various types of data or programs for driving and controlling the ultrasound diagnosis device100, input/output ultrasonic data, acquired ultrasonic images, and the like.

The input unit170may receive a user's input to control the ultrasound diagnosis device100. For example, the user's input may include an input for manipulating a button, a keypad, a mouse, a trackball, a jog switch, a knob, or the like, an input for touching a touchpad or a touch screen, a voice input, a motion input, and a bioinformation input (e.g., iris recognition or fingerprint recognition), but the present disclosure is not limited thereto.

FIG.2is a block diagram illustrating a configuration of an ultrasound diagnosis device according to another embodiment.

Referring toFIG.2, the ultrasound diagnosis device100may include a wireless probe20and a main body40. InFIG.2, components that are the same as inFIG.1are illustrated using the same reference numerals. Therefore, in describing the ultrasound diagnosis device100shown inFIG.2, a description overlapping the description inFIG.1will be omitted herein. In addition, since the reception unit115may be formed of a plurality of transducers, the “reception unit115” shown inFIG.1is illustrated as a “transducer115” inFIG.2.

The wireless probe20may include a transmission unit113, a transducer115, a reception unit117, a controller118, and a communication unit119. Although the wireless probe20has been illustrated as including both of the transmission unit113and the reception unit117inFIG.1, according to an implementation form, the wireless probe20may include only some of components of the transmission unit113and the reception unit117, and some of the components of the transmission unit113and the reception unit117may be included in the main body40. Alternatively, the wireless probe20may further include an image processing unit130.

The transducer115may include a plurality of transducers. The plurality of transducers may transmit ultrasonic signals to the object10according to a transmission signal applied from the transmission unit113. The plurality of transducers may receive ultrasonic signals reflected from the object10to form a reception signal.

The controller118controls the transmission unit113to form the transmission signal to be applied to each of the plurality of transducers in consideration of positions and focal points of the plurality of transducers.

The controller118controls the reception unit117to convert reception signals received from the transducer115in an analog-to-digital conversion manner and sum the digitally converted reception signals in consideration of the positions and the focal points of the plurality of transducers, thereby generating ultrasonic data. Alternatively, when the wireless probe20includes the image processing unit130, an ultrasonic image may be generated using the generated ultrasonic data.

The communication unit119may transmit the generated ultrasonic data or ultrasonic image to the main body40through a wireless network30in a wireless manner. In addition, the communication unit119may receive a control signal and data from the main body40.

Further, the ultrasound system100may include one or more wireless probes20according to an implementation form.

The main body40may receive ultrasonic data or an ultrasonic image from the wireless probe20. The main body40may include a controller120, an image processing unit130, a display unit140, a storage unit150, a communication unit160, and an input unit170.

FIG.3Ais a diagram illustrating an appearance of an ultrasound diagnosis device according to an embodiment.

FIG.3Bis another diagram illustrating an appearance of an ultrasound diagnosis device according to an embodiment.

FIG.3Cis another diagram illustrating an appearance of an ultrasound diagnosis device according to an embodiment.

Referring toFIGS.3A and3B, ultrasound diagnosis devices100aand100bmay each include a main display unit121and a sub display unit122. One of the main display unit121and the sub display unit122may be implemented as a touch screen. The main display unit121and the sub display unit122may display the ultrasonic image or various pieces of information processed by the ultrasound diagnosis devices100aand100b. Further, the main display unit121and the sub display unit122may be implemented as a touch screen and provide a graphical user interface (GUI) to receive data for controlling the ultrasound diagnosis devices100aand100bfrom a user. For example, the main display unit121may display the ultrasonic image, and the sub display unit122may display a control panel for controlling the ultrasonic image in the form of the GUI. The sub display unit122may receive data for controlling the displaying of the image through the control panel displayed in the form of the GUI. The ultrasound diagnosis devices100aand100bmay control, using input control data, the displaying of the ultrasonic image displayed on the main display unit121.

Referring toFIG.3B, the ultrasound diagnosis device100bmay further include a control panel165in addition to the main display unit121and the sub display unit122. The control panel165may include a button, a trackball, a jog switch, a knob, and the like, and may receive data for controlling the ultrasound diagnosis device100bfrom the user. For example, the control panel165may include a time gain compensation (TGC) button171, a freeze button172, and the like. The TGC button171is a button for setting a TGC value for each depth of the ultrasonic image. Further, when detecting the input of the freeze button172while scanning the ultrasonic image, the ultrasound diagnosis device100bmay maintain a state in which a frame image at a corresponding time point is displayed.

Meanwhile, inputs of the button, the trackball, the jog switch, the knob, and the like included in the control panel165may be provided to the GUI in the main display unit121or the sub display unit122.

Referring toFIG.3C, the ultrasound diagnosis device100cmay be implemented as a portable type. Examples of a portable ultrasound diagnosis device100cmay include a smart phone, a laptop computer, a PDA, a tablet PC, and the like including a probe and an application, but the present disclosure is not limited thereto.

The ultrasound diagnosis device100cmay include the probe20and a main body40, and the probe20may be connected to one side of the main body40in a wired or wireless manner. The main body40may include a touch screen145. The touch screen145may display the ultrasonic image, various pieces of information processed by the ultrasound diagnosis device, the GUI, and the like.

The method of displaying an ultrasonic image, the ultrasound diagnosis device, and the computer program product according to the disclosed embodiments generate a diagnostic result by analyzing the ultrasonic image to determine whether abnormality occurs in an object and provide information corresponding to the diagnostic result appearing in the ultrasonic image to a user, thereby enabling the user to accurately diagnose a patient's disease by referring to the diagnostic result to increase convenience of the user. Here, the “user” may be a doctor who diagnoses a patient's disease, a sonographer who performs an ultrasound scan on an object of a patient, or the like. In addition, the “information corresponding to the diagnostic result” refers to information displayed on the ultrasonic image corresponding to the diagnostic result, information which is the basis of the diagnostic result, a partial area in the ultrasonic image which is the basis of the diagnostic result, and a partial area in the ultrasonic image which affects the diagnostic result. Hereinafter, for convenience, the “information corresponding to the diagnostic result” will be referred to as “diagnostic information.”

As described above, the method of displaying an ultrasonic image, the ultrasound diagnosis device, and the computer program product, which are capable of increasing convenience of a user who automatically receives a diagnostic result will be described in detail with reference to the accompanying drawings.

In the disclosed embodiments, the ultrasound diagnosis device may be an electronic device capable of acquiring, processing, and/or displaying an ultrasonic image. Specifically, specifically, the ultrasound diagnosis device may refer to an electronic device capable of i) identifying a specific portion in the ultrasonic image (e.g., a portion in which a lesion occurs or the like), ii) analyzing the ultrasonic image to acquire a diagnostic result or diagnostic information, or iii) processing, generating, modifying, updating, or displaying a partial image, an entire image, or information used for diagnosis on the basis of the ultrasonic image.

Specifically, as shown inFIGS.3A to3C, the ultrasound diagnosis device according to the disclosed embodiments may be implemented as a cart type device as well as a portable type device. Examples of the portable ultrasound diagnosis device may include a picture archiving and communication system (PACS) viewer, a smart phone, a laptop computer, a PDA, a tablet PC, and the like, but the present disclosure is not limited thereto.

FIG.4is a block diagram illustrating a configuration of an ultrasound diagnosis device according to another embodiment. InFIG.4, a memory410, a processor420, and a display430may correspond to the storage unit150, the controller120, and the display unit140shown inFIG.1, respectively, and thus a description overlapping the description ofFIG.1will be omitted herein.

Referring toFIG.4, an ultrasound diagnosis device400according to the disclosed embodiments includes the memory410, the processor420, and the display430.

The memory410stores at least one instruction.

The display430displays a predetermined screen.

The processor420may execute at least one of the at least one instruction stored in the memory410to perform a predetermined operation or may control other components in the ultrasound diagnosis device400to perform a predetermined operation.

Specifically, the processor420executes at least one of the at least one instruction to identify a lesion area included in the ultrasonic image, diagnoses the lesion area to acquire a diagnostic result, displays a first area, which is at least one area that is the basis of diagnosing a lesion in the lesion area, on the lesion area of the ultrasonic image to generate a diagnostic image, and controls a user interface screen including the diagnostic image and the diagnostic result to be displayed through the display.

Here, the lesion refers to a change in the living body caused due to a disease. Specifically, a lesion may refer to all forms in which an object does not have a healthy tissue form or a case in which an object has a different state when compared with a healthy state. Specifically, the lesion may be largely classified into benign and malignant. In addition, a benign lesion may correspond to a tumor, and a malignant lesion may correspond to cancer.

In addition, in the disclosed embodiments, the “lesion area” may be a concept including not only an inner area of the lesion formed by the contour of a lesion, but also an outer area adjacent to the contour of the lesion. That is, the lesion area may mean not only an area in which the lesion itself is imaged in the ultrasonic image, but also an area in which features of the lesion are imaged.

In addition, when the lesion is present in the object, diagnostic information may be information indicating at least one among an identified lesion area, a size of the lesion, a position of the lesion, and features of the lesion (e.g., a metastasis probability, malignant risk, and the like). In addition, the diagnostic information may include standard terms for diseases defined by the society or association. For example, the diagnostic information may include information on a plurality of items included in a lexicon which is a standard term for ultrasound examination defined by the American College of Radiology (ACR).

In addition, the diagnostic result may include information indicating whether the identified lesion is malignant or benign or include a specific name of the lesion. As another example, the diagnostic result may include information indicating whether the identified lesion is a papule, a nodule, a tumor, or cancer. As another example, the diagnostic result may include information indicating a specific name of the lesion (e.g., a breast tumor, breast cancer, a thyroid tumor, a thyroid nodule, thyroid cancer, and the like).

Hereinafter, for convenience of description, a case including information indicating whether a diagnostic result is malignant or benign will be described and illustrated as an example. For example, the diagnostic result may be expressed as “Possibly benign,” “Benign,” “Possibly malignant,” “Malignant,” or the like.

In the disclosed embodiments, a first area is at least a part of the lesion area that is the basis of diagnosing the lesion and may refer to at least a part of the ultrasonic image that is the basis of diagnosing the lesion. For example, when a lesion in the object is a malignant lesion, the diagnostic result may be “malignant.” In this case, the first area may be a feature in the ultrasonic image, which is the basis of determining the corresponding lesion as malignant, or an area in which the feature in the ultrasonic image is imaged. The first area will be described in detail below with reference toFIGS.6to11.

In addition, in the disclosed embodiments, “at least one among a, b, and c” may have a meaning including all cases of including only a, including only b, including only c, including a and b, including a and c, including b and c, and including a, b, and c.

Specifically, the display430displays a predetermined screen under the control of the processor420. The display430may display a medical image (e.g., an ultrasonic image), a user interface screen, information on a patient, history information on a patient's disease, image processing information, and the like. Here, the image processing information may include an intermediate product or a final product which is generated by processing the ultrasonic image by the processor420. In addition, the “image processing” may refer to an operation of processing, generating, modifying, and/or updating an ultrasonic image.

For example, the display430may display a user interface screen including at least one among an ultrasonic image, a diagnostic image corresponding to the ultrasonic image, a diagnostic result for a lesion identified on the basis of the ultrasonic image, and information for describing the diagnostic result.

Specifically, the memory410may store at least one program required for an operation of the ultrasound diagnosis device400or at least one instruction required for executing the at least one program. In addition, the memory410may include at least one processor for performing the above-described operations.

In addition, the memory410may store at least one among an ultrasonic image, information related to the ultrasonic image, information on a patient, information on an object, and information on a testee. In addition, the memory410may store at least one among information and an image which are generated by the processor420. In addition, the memory410may store at least one among an image, data, and information which are received from an external electronic device (not shown).

The processor420may generally control an operation of the ultrasound diagnosis device400. Specifically, the processor420may execute at least one instruction to perform a predetermined operation or control the predetermined operation to be performed. Here, the instruction executed by the processor420may be at least one of the at least one instruction stored in the memory410.

In addition, the processor420may be implemented in the form of one processor or in the form in which a plurality of processors are combined.

In addition, the processor420may include an internal memory (not shown) and at least one processor (not shown) for executing at least one stored program. Specifically, the internal memory (not shown) of the processor420may store one or more instructions. In addition, the at least one processor (not shown) included in the processor420may execute at least one among the one or more instructions stored in the internal memory (not shown) of the processor420to perform a predetermined operation.

Specifically, the processor420may include a random access memory (RAM) (not shown) used as a storage area for storing signals or data which are input from the outside of the ultrasound diagnosis device400or used as a storage area corresponding to various tasks performed by the ultrasound diagnosis device400, a read only memory (ROM) (not shown) in which a control program and/or a plurality of instructions for controlling the ultrasound diagnosis device400are stored, and at least one processor (not shown). The processor (not shown) may include a graphic processor (Graphic Processing Unit, not shown) for graphics processing corresponding to video data. The processor (not shown) may be implemented as a system on chip (SoC) in which a core (not shown) and the GPU (not shown) are integrated. A processor (not shown) may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof.

For convenience of description, in the disclosed embodiment, a case in which the processor420executes an instruction stored in the memory410to perform or control a predetermined operation will be described as an example.

FIG.5is a block diagram illustrating a configuration of an ultrasound diagnosis device according to another embodiment. InFIG.5, components that are the same as inFIG.4are illustrated using the same reference numerals. Therefore, in describing an ultrasound diagnosis device500shown inFIG.5, a description overlapping the description inFIG.4will be omitted herein.

When compared with the ultrasound diagnosis device400, the ultrasound diagnosis device500may further include at least one among a data acquisition unit440, an image processing unit450, a user interface460, and a communication unit470. The data acquisition unit440, the image processing unit450, the user interface460, and the communication unit470may be identically correspond to the ultrasonic transceiver110, the image processing unit130, the input unit170, and the communication unit160ofFIG.1, respectively. Therefore, in describing the ultrasound diagnosis device500, a description overlapping the description inFIG.1will be omitted herein.

The data acquisition unit440may acquire ultrasonic data. Specifically, the data acquisition unit440may receive an ultrasound echo signal, which is reflected from the object, through the probe20under the control of the processor420.

Specifically, the data acquisition unit440may directly acquire raw data for generating an ultrasonic image under the control of the processor420. In this case, the data acquisition unit440may correspond to the ultrasonic transceiver110. Here, the raw data may refer to original data corresponding to the ultrasound echo signal acquired by being reflected from the object or refer to data before processing.

Then, the processor420may control to generate an ultrasonic image corresponding to the ultrasonic echo signal received through the data acquisition unit440. Specifically, the processor420may control the image processing unit450to generate the ultrasonic image using the ultrasound echo signal.

In addition, the data acquisition unit440may receive an ultrasonic image from an external electronic device (not shown), such as an external ultrasound diagnosis device (not shown), under the control of the processor420. Specifically, the data acquisition unit440may be connected to an external electronic device (not shown) through a wired/wireless communication network and may receive the ultrasonic image or raw data for generating the ultrasonic image, which are transmitted from the external electronic device (not shown) under the control of the processor420. When the data acquisition unit440acquires the raw data, the processor420may control the image processing unit450to generate the ultrasonic image using the raw data received through the data acquisition unit440.

When the data acquisition unit440receives the ultrasonic image or the raw data from the external electronic device (not shown), the data acquisition unit440may include the communication unit470. That is, the communication unit470may be the data acquisition unit440itself, and the communication unit470may be implemented in the form of being included in the data acquisition unit440.

The image processing unit450may perform at least one operation of generating an ultrasonic image and processing the ultrasonic image. In addition, the image processing unit450may be implemented in the form of being included in the processor420.

For example, when the processor420includes a plurality of processors, any one among the plurality of processors included in the processor420may be an image processor for image processing.

The user interface460may receive predetermined data or a predetermined command from a user. The user interface460may correspond to the input unit170ofFIG.1. In addition, the user interface460may be formed as a touch screen or a touch pad which is integrally formed with the display430. As another example, the user interface460may include a user input device such as a button, a key pad, a mouse, a trackball, a jog switch, and a knob. In addition, the user interface460may include a microphone, a motion detection sensor, and a biometric information detection sensor for receiving a voice input, a motion input, and a biometric information input (e.g., iris recognition, fingerprint recognition, and the like).

Thus, the user interface460may receive inputs manipulating buttons, keyboards, mouses, trackballs, jog switches, and knobs, an input of touching a touch pad or a touch screen, a voice input, a motion input, and/or a biometric information input (e.g., iris recognition, fingerprint recognition, and the like).

In the disclosed embodiment, the processor420may identify the lesion area on the basis of a user input received through the user interface460. For example, the ultrasound diagnosis device500may display the ultrasonic image through the display430. The user may look at the ultrasonic image and input an input for selecting a predetermined position or a predetermined area corresponding to a portion suspected of a lesion in the ultrasonic image through the user interface460. Then, on the basis of the received user input, the processor420may analyze the ultrasonic image around or near the predetermined position or the predetermined area selected by the user input or analyze the ultrasonic image within a predetermined range based on the selected position or the predetermined area. Thus, according to the analysis result, it is possible to identify a contour of the lesion and the lesion area formed by the contour of the lesion.

In addition, in the disclosed embodiment, the processor420may automatically detect the lesion area included in the ultrasonic image through a computer-based image processing technology. In the above-described example, the user may look at the ultrasonic image and input an input for selecting a predetermined position or a predetermined area corresponding to a portion suspected of a lesion in the ultrasonic image through the user interface460. Then, the processor420may analyze at least a partial area of the ultrasonic image corresponding to a position or an area selected through the user input using a computer-based automatic diagnosis technology. In addition, as the analysis result, the processor420may precisely extract the margin of the lesion or the lesion area formed by the margin of the lesion.

Here, the computer-based image processing technology may include a diagnosis technology based on machine learning, and the like.

Here, the machine learning may be performed through a CAD system which determines and detects whether abnormality or a lesion occurs in an object through a computer operation, a statistical machine learning based on data, or an artificial intelligence system which performs machine learning according to an artificial intelligence technology. In the disclosed embodiment, the processor420may analyze the ultrasonic image using a computer-based image processing technology and acquire desired information (e.g., the lesion area, information on the lesion, a diagnostic result, and the like) as the analysis result.

Here, unlike the existing rule-based smart system, the artificial intelligence (AI) system is a system in which a machine learns and determines by itself and generates a result desired by the user. The AI technology includes machine learning (deep learning) and element technologies using the machine learning. The machine learning is an algorithm technology which categorizes/learns features of input data by itself, and the element technology is a technology which utilizes a machine learning algorithm such as deep learning and includes technical fields such as linguistic understanding, visual understanding, deduction/prediction, knowledge expression, and motion control.

In the disclosed embodiment, the processor420may use the machine learning that is the aforementioned computer-based image processing technology in identifying the lesion area, acquiring the diagnostic result, acquiring the diagnostic information, and/or establishing a criterion for identifying the lesion.

Specifically, image processing through the AI technology may be performed through an operation based on a neural network. Specifically, an operation based on a neural network such as a deep neural network (DNN) may be used. In addition, an operation of the DNN may include an operation of a convolutional neural network (CNN). An operation of processing the ultrasonic image through an AI technology will be described in detail below with reference toFIG.29.

Hereinafter, operations performed in the ultrasound diagnosis device100,400, or500according to the disclosed embodiment will be described in detail with reference toFIGS.6to32. In addition, for convenience of description, the operations of the disclosed embodiment will be described below using the ultrasound diagnosis device400ofFIG.4as an example.

FIG.6is a diagram illustrating an ultrasonic image obtained in an embodiment.

Referring toFIG.6, an ultrasonic image600acquired by the ultrasound diagnosis device400is shown. Specifically,FIG.6illustrates the ultrasonic image600corresponding to a breast portion acquired through a breast ultrasound scan as an example.

The processor420identifies a lesion area610included in the ultrasonic image600. Specifically, the processor420may analyze the ultrasonic image600using an automatic diagnosis technology such as a CAD technology or an AI technology. In addition, as the analysis result, the processor420may automatically detect the lesion area610included in the ultrasonic image600. Hereinafter, operations of identifying the lesion area610and analyzing the lesion area610to acquire at least one of a diagnostic result and diagnostic information may be performed using a CAD technology.

For example, the user may look at the ultrasonic image600and select an area suspected of the lesion or one position or a predetermined area in an area adjacent to a suspected portion in the ultrasonic image600through a user interface460. Specifically, the user may point, touch, or click on one position611in the lesion area610through the user interface360. Then, the processor420may analyze an area surrounding the position or area selected through the user input using a CAD technology or an AI technology to accurately detect the lesion area610.

In addition, the processor420may analyze the lesion area610using a CAD technology or an AI technology to acquire a diagnostic result.

In addition, in order to allow the user to easily recognize the detected lesion area610, the processor420may process a contour615of the lesion area610to generate a diagnostic image (not shown) so as to allow the contour615of the lesion area610to be clearly displayed. For example, in order to allow the user to easily recognize the contour615of the lesion area610, the diagnostic image (not shown) may be an image which is generated by marking a line of a predetermined color on the contour615of the lesion area610in the ultrasonic image600. In addition, various processes of allowing the user to easily recognize the lesion area610may exist. As another example, the diagnostic image (not shown) may be generated through a process of adjusting transparency of the lesion area610, marking the lesion area610with a specific color or pattern, or marking the contour615of the lesion area610with an outline such as a dotted line or a solid line.

Then, the processor420displays a first area on the lesion area610of the ultrasonic image600to generate the diagnostic image (not shown). Here, the first area is at least a part of the lesion area that is the basis of diagnosing the lesion and may refer to at least a part of the ultrasonic image that is the basis of diagnosing the lesion.

When the lesion is distinguished as malignant or benign, a feature of the lesion classified as malignant is different from a feature of the lesion classified as benign. When the lesion is distinguished as malignant or benign, a criterion for distinguishing the lesion may be set on the basis of at least one among a CAD system, an AI system, and a user setting. Various types of lesions exist, and the lesions may be distinguished according to various criteria. Thus, the lesion discrimination criterion may be optimized and set by an experiment or machine learning.

Specifically, the lesion area610is the basis of analyzing the lesion, and the diagnostic result may be determined according to contents contained in a periphery of the lesion area610and an inner area thereof. For example, the lesion may be distinguished as a specific lesion on the basis of a result of analyzing at least one among a shape of the lesion area, an orientation of the lesion, a margin of the lesion, an echo feature exhibiting in the lesion area, and a posterior feature of the lesion area.

In diagnosing the lesion using a CAD technology and the like, it is important to accurately extract the lesion area. In addition, the lesion area may be accurately extracted only when image accuracy of the lesion area610, which is an area in which the lesion is imaged in the ultrasonic image600, and image accuracy of a surrounding area of the lesion area610are high.

However, according to a specific shape of the lesion, quality of the ultrasonic image, the product specifications of the ultrasound diagnosis device which performs an ultrasound scan, and a setting applied during the ultrasound scan of the ultrasound diagnosis device, there may occur a case in which an accurate extraction of the lesion area is difficult.

In particular, when the contour of the identified lesion area is changed, the shape of the lesion area, the orientation of the lesion, the margin feature of the lesion, and the like may be changed. Accordingly, the diagnostic result of the lesion may also be varied. Therefore, when the lesion area is not accurately extracted and thus the diagnostic result is not accurately derived, reliability of the diagnostic result may be degraded.

The existing ultrasound diagnosis device, which provides a diagnostic result using a CAD system and the like, provides only a diagnostic result without describing a process of deriving the diagnostic result. Accordingly, even when there occurs a case in which an erroneous diagnostic result is provided or a case in which determination is ambiguous in deriving the diagnostic result, the user does not recognize such errors. Thus, when the user trusts the erroneous diagnostic result to finally derive the diagnostic result, a patient's disease may be misdiagnosed.

In addition, the provision of the erroneous diagnostic result does not particularly help the user in diagnosing the object and may confuse the user in diagnosing the object.

In the embodiments of the present disclosure, in the ultrasound diagnosis device400which identifies a lesion area by analyzing an ultrasonic image and provides a diagnostic result for the lesion, the diagnostic image in which an area (specifically, the first area) that is the basis of the diagnostic result is displayed is provided to the user, thereby allowing the user to trust the diagnostic result more and to more easily diagnose the object by referring to the diagnostic result and the first area.

For example, when a shape of the lesion area has an irregular or variable shape, a margin of the lesion area becomes a spiculated pattern, and an echo feature has a low-level marked hyperechogenic feature, a breast lesion is more likely to be diagnosed as malignant. Since the lesion area610included in the ultrasonic image600shown inFIG.6has the above-described feature of the malignant lesion, the processor420may generate a diagnostic result such as “Malignant” or “Possibly Malignant” as the analysis result of the lesion area610of the ultrasonic image600.

In the above-described example, the basis of diagnosing the lesion included in the ultrasonic image600as malignant may be that the shape of the lesion area610has an irregular shape instead of an oval or an ellipsoid and a spiculated pattern is present in the margin of the lesion area. Accordingly, the shape of the contour615of the lesion area610and margin areas620and630, each having a spiculated pattern in the lesion area610, may be the basis of the diagnostic result (specifically, the lesion is benign).

Therefore, in the disclosed embodiment, since the diagnostic image, in which the areas that are the basis for diagnosing the lesion area610as malignant (e.g.,620,615which is an area in which a contour is displayed, or630) are displayed, is provided to the user, the user may diagnose the object quickly and easily.

In the embodiments of the present disclosure, the first area, which is at least one area that is the basis of diagnosing the lesion, may be displayed to be distinguished from other area in the diagnostic image. Specifically, the processor420may distinguish and display the first area and other areas in the diagnostic image using at least one among a specific color, a symbol, transparency, an echo, a mark, and a line having a specific pattern.

FIG.7is a diagram illustrating a user interface screen including a diagnostic image provided in an embodiment. Specifically,FIG.7illustrates a user interface screen700output from the ultrasound diagnosis device400. For convenience of description, inFIGS.8,1021, and23to28includingFIG.7, a lesion area (e.g.,720) included in a diagnostic image (e.g.,710) is simplified and schematically illustrated. Here, a contour of the lesion area720may correspond to the contour615of the lesion area ofFIG.6identically.

Referring toFIG.7, the user interface screen700may include the diagnostic image710and a partial screen750indicating a diagnostic result. Hereinafter, for convenience of description, the partial screen750indicating the diagnostic result included in the user interface screen700will be referred to as a “diagnostic result750.”

Referring toFIG.7, a portion that is the basis of determining a lesion imaged in the diagnostic image710as “Malignant” may be referred to as a portion having a spiculated pattern of the margin of the lesion area720. Accordingly, the diagnostic image710may be an image in which an area730having a spiculated pattern is displayed as a first area in the margin of the lesion area720.

For example, the processor420may display the first area (e.g.,730) to be distinguished from other areas. InFIG.7, a case in which the first area is displayed using an area indication line is illustrated as an example. Accordingly, the user may look at the diagnostic image710, which is an ultrasonic image including the lesion, to easily recognize a feature of the lesion (specifically, a margin having a spiculated pattern) imaged in the first area (e.g.,730). Thus, the user may quickly and easily determine whether the diagnostic result (e.g., “Malignant”) provided by the ultrasound diagnosis device is correct.

In addition, inFIG.7, a case in which one area is displayed as the first area in the diagnostic image710is illustrated as an example. However, in the disclosed embodiment, the first area may be present as a plurality of areas in the diagnostic image710. That is, in the disclosed embodiment, the “first area” merely refers to the area that is the basis of diagnosing the lesion and does not refer to the number of area being one.

In addition, when a plurality of features indicating a feature of the lesion exist in the lesion area (e.g.,610or701), the first area may be an area indicating at least one among the plurality of features. Here, the plurality of features are features of the lesion for distinguishing a type of a lesion and include the above-described shape of the lesion area, the orientation of the lesion, the margin of the lesion, the echo feature appearing in the lesion area, and a posterior feature of the lesion area.

Specifically, the plurality of above-described features may correspond to a plurality of items included in a lexicon which is a standard term for ultrasound examination defined by the ACR.

Accordingly, the first area may be an area indicating at least one feature among the plurality of above-described features.

Specifically, inFIG.6, the displayed first areas620and630may be areas indicating the margin of the lesion among the above-described features. In addition, inFIG.6, the displayed contour line615may be an area indicating the shape of the lesion area among the above-described features.

In the disclosed embodiment, in the diagnostic image710, at least one area indicating any one among the plurality of above-described features may be displayed as the first area.

In addition, in the diagnostic image710, a plurality of areas indicating two or more of the plurality of above-described features may be displayed as the first area. Here, when the plurality of areas indicating the plurality of features are displayed in the diagnostic image710, the plurality of areas may be displayed in the diagnostic image710to allow different features to be distinguished and displayed. Specifically, when the plurality of features indicating the lesion area exist, the processor420may display at least one area in the lesion area using at least one among different colors, marks, symbols, transparency, echo, a type of dotted line, and a thickness of the dotted line, thereby distinguishing different features from each other among the plurality of features. In addition, the processor420may control the diagnostic image in which the at least one area is displayed to be displayed.

For example, when the ultrasonic image600shown inFIG.6is generated as a diagnostic image, the areas620and630which are areas indicating the margin feature of the lesion may be marked with an indication line having a red color, and the contour line615indicating the shape feature of the lesion may be marked with an indication line having a blue color.

FIG.8is a diagram illustrating a user interface screen including another diagnostic image provided in an embodiment. Specifically,FIG.8illustrates a user interface screen700output from the ultrasound diagnosis device400. In the user interface screen800shown inFIG.8, the same components as in the user interface screen700shown inFIG.7are illustrated using the same reference numerals.

In the disclosed embodiment, the processor420may display diagnostic information, which is information on the feature of the lesion area (e.g., the margin feature of the lesion area) appearing in the first area (e.g.,730), in the ultrasonic image, thereby generating a diagnostic image810.

Here, the diagnostic information may be information indicating the feature of the lesion area appearing in the first area or information that is the basis of deriving a diagnostic result.

For example, since the feature of the lesion area appearing in the first area730becomes the margin feature of the lesion area, diagnostic information835may be “Margin: spiculated” which is information specifically indicating the margin feature. In addition, the diagnostic information may be “malignant determination basis: Margin-spiculated” which is information that is the basis of deriving malignant determination that is the diagnostic result.

In addition, the diagnostic information may be displayed in an area corresponding to the first area in the diagnostic image (e.g., an area adjacent to the first area that does not interfere with user's observation of the first area).

In the above-described example, the processor420may generate the diagnostic image810by displaying “Margin: spiculated” or “malignant determination basis: Margin—spiculated” in the area835adjacent to the first area730.

FIG.9is another diagram illustrating an ultrasonic image obtained in an embodiment. Referring toFIG.9, an ultrasonic image900acquired by the ultrasound diagnosis device400is shown. Specifically,FIG.9illustrates the ultrasonic image900corresponding to a breast portion acquired through a breast ultrasound scan as an example.

In the case of a breast lesion, a shape of a lesion area is an oval or an ellipsoid, a margin of the lesion area becomes a smooth pattern, and when an echo feature has a hyperechogenic feature, the breast lesion is more likely to be diagnosed as benign. Since a lesion area910included in the ultrasonic image900shown inFIG.9has the above-described feature of the benign lesion, the processor420may generate a diagnostic result such as “Benign” or “Possibly Benign” as the analysis result of the lesion area610of the ultrasonic image600.

In the above-described example, the basis of diagnosing the lesion included in the ultrasonic image900as benign may be that the shape of the lesion area910is an oval or an ellipsoid and the margin of the lesion area has a smooth pattern. Accordingly, a shape920of the lesion area910and a margin area930having a smooth pattern in the lesion area910may be at least some areas in the ultrasonic image900, which are the basis of the diagnostic result (specifically, the lesion is benign). Here, a line920indicating the shape920of the lesion area910may be displayed to overlap the contour915of the lesion area910or may be displayed in an area adjacent to the lesion area910.

In the disclosed embodiment, the diagnostic image in which the areas (e.g.,920, and930) that are the basis of diagnosing the lesion area910as benign are displayed is provided to the user, thereby allowing the user to trust the diagnostic result more and to diagnose the object more easily by referring to the diagnostic result and the first area.

FIG.10is a diagram illustrating a user interface screen including another diagnostic image provided in an embodiment. Specifically,FIG.7illustrates a user interface screen1000output from the ultrasound diagnosis device400. In addition, similar toFIGS.8and9,FIG.10schematically illustrates a lesion area1020included in a diagnostic image110. Here, a contour of the lesion area1020may correspond to the contour915of the lesion area ofFIG.9identically.

Referring toFIG.10, the user interface screen1000may include the diagnostic image1010and a diagnostic result1050.

A portion that is the basis of determining a lesion imaged in the diagnostic image1010as “Benign” may be a portion having a smooth pattern of the margin of the lesion area1020. Accordingly, the diagnostic image1010may be an image in which an area730having a smooth pattern is displayed as a first area in the margin of the lesion area1020. The user may look at the diagnostic image1010, which is an ultrasonic image including the lesion, to easily recognize a feature of the lesion (specifically, the margin having the smooth pattern) imaged in a first area (e.g.,1030). Thus, the user may quickly and easily determine whether the diagnostic result (e.g., “Benign”) provided by the ultrasound diagnosis device is correct.

FIG.11is a diagram illustrating a user interface screen including another diagnostic image provided in an embodiment. Specifically,FIG.11illustrates a user interface screen1100output from the ultrasound diagnosis device400. In the user interface screen1100shown inFIG.11, the same components as in the user interface screen1000shown inFIG.10are illustrated using the same reference numerals.

In the disclosed embodiment, the processor420may display diagnostic information1135, which is information on the feature of the lesion area (e.g., a margin feature of the lesion area) appearing in the first area (e.g.,1030), in the ultrasonic image, thereby generating a diagnostic image1110.

For example, since the feature of the lesion area appearing in the first area1030becomes the margin feature of the lesion area, diagnostic information1135may be “Margin: smooth” which is information specifically indicating the margin feature. In addition, the diagnostic information may be “benign determination basis: Margin-smooth” which is information that is the basis of deriving malignant determination that is the diagnostic result.

FIG.12is another diagram illustrating a user interface screen including another diagnostic image provided in an embodiment.

In the disclosed embodiment, as the analysis result of the lesion, the processor420may determine whether the diagnostic result has accuracy that is greater than or equal to a limit value, and when the diagnostic result having the accuracy that is greater than or equal to a limit value is not present, the processor420may derive at least two diagnostic results.

For example, when the diagnostic result, which is acquired as the analysis result of the lesion, has the accuracy of 90% or more, the processor420may output one diagnostic result. When the lesion610shown inFIG.6is determined as malignant with the accuracy of 90%, the processor420may determine the diagnostic result of the lesion610as “malignant.”

In contrast, when the diagnostic result, which is acquired as the analysis result of the lesion, has the accuracy that is less than the limit value (e.g., 90%), the processor420may output a plurality of diagnostic results. For example, the diagnostic result1050with respect to the lesion1220may be displayed together with the diagnostic accuracy for each of the plurality of diagnostic results as in the user interface1200. Also, when the plurality of diagnostic results are output, the processor420may output the plurality of diagnostic results in the order of high-to-low accuracy thereof.

An example in which, as the analysis result of the lesion area1220included in the ultrasonic image, a probability that the lesion1220is determined as being benign is 70% and a probability that the lesion1220is determined as being malignant is 30% will be described. In addition, an example in which the processor420sets the limit value corresponding to the accuracy of the diagnostic result to 90%. In this case, the diagnostic result1050with respect to the lesion1220may be displayed as “Benign: 30% and Malignant: 70%.”

In addition, a diagnostic image1210may be an image in which both of the first area (e.g.,1030) that is the basis of the benign determination and the first area (e.g.,730) that is the basis of the malignant determination are displayed. In addition, similar toFIGS.9and11, the diagnostic image1210may be an image in which the pieces of diagnostic information (e.g.,835and1135) corresponding to the first areas are additionally displayed.

FIG.13is a diagram illustrating an example of the user interface screen provided in the embodiment.

In the disclosed embodiment, the processor420may generate a user interface screen1400including a diagnostic image1310, a diagnostic result1330, and a list1302indicating features of a lesion area. In addition, the display430may display the user interface screen1400under the control of the processor420.

In addition, the diagnostic image1310and the diagnostic result1330may be formed as one partial screen1301. Specifically, the processor420may display the diagnostic result1330to overlap in a partial area of the diagnostic image1310, thereby generating the diagnostic image1310. InFIG.13, the diagnostic result1330is classified into benign and malignant.

In addition, the list1302includes items corresponding to the features indicating the lesion. Here, each of the plurality of features is the feature of the lesion for distinguishing a type of a lesion and includes the above-described shape of the lesion area, the orientation of the lesion, the margin of the lesion, the echo feature appearing in the lesion area, and a posterior feature of the lesion area.

Specifically, the plurality of above-described features may correspond to a plurality of items included in a lexicon which is a standard term for ultrasound examination defined by the ACR. In this case, the list1302may include a plurality of items1360included in the lexicon.

In the drawings described and illustrated below, includingFIG.13, an example in which the list1302indicating the features of the lesion area is formed as a lexicon will be illustrated and described.

In an embodiment of the present disclosure, when a plurality of partial areas corresponding to the plurality of features are present, the processor420may generate a diagnostic image in which the plurality of partial areas are displayed to be distinguished from each other so as to distinguish the plurality of features from each other.

Alternatively, the processor420may generate the diagnostic image1320such that the first area corresponding to an item selected by the user from among the plurality of items1360included in the list1302is included in the diagnostic image1320.

Specifically, in response to a user input for selecting any one among the plurality of items included in the list1302, the processor420may display at least one partial area, which indicates a feature corresponding to the selected item, as the first area in the ultrasonic image, thereby generating the diagnostic image. Then, the processor420may update, generate, and display the user interface screen1400to reflect the generated diagnostic image.

When the ultrasound diagnosis device400displays the user interface screen1300, the user may look at the user interface screen1300and may want to identify a partial area on the ultrasonic image in which any one among the plurality of features that are the basis of the diagnostic result1330appears. In this case, the user may select any one among the plurality of features that are the basis of the diagnostic result1330through the user interface460. Specifically, the user may select an item corresponding to any one among the plurality of features included in the list1302. Accordingly, the user interface460may receive a user input for selecting one item included in the list1302. Then, on the basis of the user input received through the user interface460, the processor420may identify a feature corresponding to the selected item and generate the diagnostic image to allow an area (specifically, the first area) indicating the identified feature to be displayed in the ultrasonic image.

That is, on the basis of the user input received through the user interface460, the ultrasound diagnosis device400may output a user interface screen including a diagnostic image generated by displaying an area (specifically, the first area), which indicates a feature corresponding to the selected item in the ultrasonic image. Here, an area indicating a predetermined feature (specifically, the first area that is a partial area in the ultrasonic image) may mean an area that is the basis of determination or determining the predetermined feature. For example, when the shape feature of the lesion area is determined as an oval, the first area may be at least a partial contour area having a curvature corresponding to an ellipse among the lesion areas.

As described above, when the diagnostic image in which the first area indicating the feature selected on the basis of the user input is provided, the user may easily identify a partial area in the ultrasonic image in which the feature to be checked is displayed. Accordingly, convenience and ease for the user can be increased in diagnosing the lesion by the user.

InFIGS.14to19to be described below, components that are the same as inFIG.13are illustrated using the same reference numerals.

FIG.14is a diagram illustrating another example of the user interface screen provided in the embodiment.

Referring toFIG.14, a user interface screen1400may correspond to the user interface screen1300.

A user input in which the user looks at the user interface screen1300and selects a “Shape” item included in the list1302may be input to the ultrasound diagnosis device400. Then, the processor420may identify a partial area1410indicating a “shape” feature selected in response to the reception of the user input and control to output the user interface screen1400including a diagnostic image1401in which an identified partial area1410is displayed as a first area. In addition, although a case in which a partial area that is the basis of determining a predetermined feature is one has been illustrated as an example inFIG.154, it is obvious that a plurality of partial areas that are the basis of determining the predetermined feature may exist.

In addition, in the user interface screen1400, a selected item1411in the list1302may be displayed to be distinguished from unselected items (e.g., an orientation item, a margin item, an echo item, and a posterior item). For example, the processor420may generate the diagnostic image1401in which the selected item1411in the list1302is displayed to be distinguished from the unselected items (e.g., the orientation item, the margin item, the echo item, and the posterior item) using at least one among different colors, symbols, transparency, echo, marks, and lines of specific patterns.

Also, the processor420may generate a diagnostic image in which the diagnostic information (e.g.,835or1135) specifically indicating the feature of the selected item is displayed in an area corresponding to the partial area1410. For example, when the shape feature of the lesion area appearing in the partial area1410is an “oval,” the processor420may generate a diagnostic image (not shown) in which diagnostic information written as “Shape-Oval” is displayed in an area corresponding to the partial area1410.

FIG.15is a diagram illustrating another example of the user interface screen provided in the embodiment. Referring toFIG.15, a user interface screen1500may correspond to the user interface screen1300.

A user input in which the user looks at the user interface screen1500and selects an “Orientation” item1511included in the list1302may be input to the ultrasound diagnosis device400. Then, the processor420may identify a partial area1510indicating an “orientation” feature selected in response to the reception of the user input and control to output the user interface screen1500including a diagnostic image1501in which an identified partial area1510is displayed as a first area.

Also, the processor420may generate a diagnostic image in which the diagnostic information (e.g.,835or1135) specifically indicating the feature of the selected item is displayed in an area corresponding to the partial area1510. For example, when the orientation feature of the lesion area appearing in the partial area1510is “Parallel,” the processor420may generate a diagnostic image (not shown) in which diagnostic information written as “Orientation-Parallel” is displayed in an area corresponding to the partial area1510.

FIG.16is a diagram illustrating another example of the user interface screen provided in the embodiment. Referring toFIG.16, a user interface screen1600may correspond to the user interface screen1300.

A user input in which the user looks at the user interface screen1600and selects a “Margin” item1611included in the list1302may be input to the ultrasound diagnosis device400. Then, the processor420may identify partial areas1610and1620indicating a “Margin” feature selected in response to the reception of the user input and control to output the user interface screen1600including a diagnostic image1601in which the identified partial area1610and1620are displayed as a first area.

In addition, the processor420may generate a diagnostic image in which the diagnostic information (e.g.,835or1135) specifically indicating the feature of the selected item is displayed in areas corresponding to the partial areas1610and1620. For example, when the margin feature of the lesion area appearing in the partial areas1610and1620is “smooth,” the processor420may generate a diagnostic image (not shown) in which diagnostic information written as “Margin-smooth” is displayed in the areas corresponding to the partial areas1610and1620.

FIG.17is a diagram illustrating another example of the user interface screen provided in the embodiment. Referring toFIG.17, a user interface screen1700may correspond to the user interface screen1300.

A user input in which the user looks at the user interface screen1700and selects an “Echo” item1711included in the list1302may be input to the ultrasound diagnosis device400. Then, the processor420may identify partial areas1710and1720indicating an “echo” feature selected in response to the reception of the user input and control to output the user interface screen1700including a diagnostic image1701in which the identified partial area1710and1720are displayed as a first area.

In addition, the processor420may generate a diagnostic image in which the diagnostic information (e.g.,835or1135) specifically indicating the feature of the selected item is displayed in areas corresponding to the partial areas1710and1720. For example, when the echo feature of the lesion area appearing in the partial areas1710and1720is “hyperechoic,” the processor420may generate a diagnostic image (not shown) in which diagnostic information written as “Echo-hyperechoic” is displayed in the areas corresponding to the partial areas1710and1720.

FIG.18is a diagram illustrating another example of the user interface screen provided in the embodiment. Referring toFIG.18, a user interface screen1800may correspond to the user interface screen1300.

A user input in which the user looks at the user interface screen1800and selects a “Posterior” item1811included in the list1302may be input to the ultrasound diagnosis device400. Then, the processor420may identify a partial area1810indicating a “posterior” feature selected in response to the reception of the user input and control to output the user interface screen1800including a diagnostic image1801in which an identified partial area1810is displayed as a first area.

For example, when the object is the abdomen including the gallbladder, a lesion area is identified, and an outer area of the identified lesion area (specifically, a posterior area of the lesion area) is imaged very darkly within the ultrasonic image, it may be determined that a stone is present as a lesion in the gallbladder. In the above example, when a “posterior” feature is selected, the processor420may generate a diagnostic image in which a very darkly imaged area of the outside area of the identified lesion area is displayed as a first area.

Also, the processor420may generate a diagnostic image in which the diagnostic information (e.g.,835or1135) specifically indicating the feature of the selected item is displayed in an area corresponding to the partial area1810. For example, when the posterior feature of the lesion area appearing in the partial area1810is “No Posterior Findings,” the processor420may generate a diagnostic image (not shown) in which diagnostic information written as “No Posterior Findings” is displayed in an area corresponding to the partial area1810.

FIG.19is a diagram illustrating another example of the user interface screens provided in the embodiment. InFIG.19, components that are the same as inFIGS.13to18are illustrated using the same reference numerals.

In the disclosed embodiment, the processor420may generate and display a user interface screen such that a plurality of diagnostic images corresponding to a plurality of features, each indicating a lesion result, are sequentially displayed.

Specifically, the processor420may generate and display the diagnostic image such that features corresponding to items included in the list1302are sequentially displayed. For example, the processor420may control the ultrasound diagnosis device400to sequentially display the diagnostic image1401shown inFIG.14, the diagnostic image1501shown inFIG.15, the diagnostic image1601shown inFIG.16, the diagnostic image1701shown inFIG.17, and the diagnostic image1801shown inFIG.18.

In addition, the display order among the diagnostic image1401, the diagnostic image1501, the diagnostic image1601, the diagnostic image1701, and the diagnostic image1801may be determined on the basis of a user setting, a proprietary setting of the processor420, importance between the plurality of features, or influences of the plurality of features on the diagnostic result.

FIG.20is a diagram illustrating another example of the user interface screen provided in the embodiment. Referring toFIG.20, a user interface screen2000may correspond to the user interface screen1300shown inFIG.13. As described with reference toFIG.13, the user may select at least two items from among the plurality of items corresponding to the plurality of features included in the list1302. Accordingly, the processor420may control to generate and display a user interface screen2000including a diagnostic image2010, in which partial areas indicating the plurality of features are displayed, and a diagnostic result2050.

For example, when the margin item and the shape item included in the list1302are selected through a user input, the processor420may control to generate and display the diagnostic image2010in which a partial area2040that is the basis of determining a margin feature is displayed to be distinguished from a partial area2030that is the basis of determining a shape feature. In addition, the processor420may generate the diagnostic image2010in which diagnostic information (e.g.,2041) is displayed to overlap an area corresponding to the partial area (e.g.,2040). In addition, inFIG.20, a case in which the partial area2040and the partial area2030are distinguished through patterns of different dotted lines is illustrated as an example.

In addition, when the user does not select a specific item (or a feature), the processor420may generate the diagnostic image (e.g.,2010) in which partial areas indicating two or more of the features among the plurality of features are displayed on the basis of a user setting, a proprietary setting of the processor420, importance among a plurality of features, or influences of the plurality of features on the diagnostic result. Specifically, the processor420may display the partial areas in the diagnostic image using at least one among a specific color, a symbol, transparency, an echo, a mark, and a line having a specific pattern to distinguish the plurality of features from each other.

For example, the processor420displays a partial area indicating the shape feature of a lesion area with a red dotted line, displays a partial area indicating an orientation feature of a lesion with a blue dotted line, displays a partial area indicating a margin feature of the lesion with a green dotted line, displays a partial area indicating an echo feature appearing in the lesion area with a yellow dotted line, and displays a partial area indicating a posterior feature of the lesion area with a purple dotted line so that it is possible to generate a diagnostic image in which different features are distinguished from each other.

FIG.21is a diagram illustrating another example of the user interface screen provided in the embodiment. InFIG.21, components that are the same as inFIG.20are illustrated using the same reference numerals.

In the disclosed embodiment, in a list2102, a plurality of items may include information specifically representing features corresponding to the plurality of items. For example, a shape item may include information of “irregular,” which is a feature specifically describing the shape feature, and a margin item may include information of “spiculated.”

In addition, the processor420may control each of the plurality of items included in the list2102to be displayed to match at least one partial area corresponding thereto. Specifically, the processor420may display each of the plurality of items to match at least one partial area corresponding thereto using at least one among different colors, marks, symbols, transparency, echo, types of dotted lines, and thicknesses of the dotted lines.

For example, a case in which the partial image2030is a partial image that is the basis of determining the shape feature, and the partial image2040is a partial image that is the basis of determining the margin feature will be described as an example. In this case, the processor420may control the partial image2030and the shape item in the list2102to be displayed with at least one among the same color, the same mark, the same symbol, the same transparency, the same echo, the same type of dotted line, and the same thickness of the dotted line. In addition, the processor420may control the partial image2040and the margin item in the list2102to be displayed with at least one among the same color, the same mark, the same symbol, the same transparency, the same echo, the same type of dotted line, and the same thickness of the dotted line. In this case, the shape item and the margin item may be displayed with at least one among colors, marks, symbols, transparency, echoes, types of dotted lines, and thicknesses of the dotted lines, which are distinguished from each other.

In addition, the diagnostic result2050may be determined and displayed as malignant on the basis of the fact that the shape of the lesion area is irregular and the margin thereof is spiculated.

FIG.22is a diagram illustrating another example of the user interface screen provided in the embodiment.

Referring toFIG.22, the processor420may generate a user interface screen1400including a diagnostic image2210, a diagnostic result2250, and a list2202indicating features of a lesion area.

In the disclosed embodiment, when a predetermined feature included in the list2202is changed and thus a diagnostic result is changed, the processor420may distinguish the predetermined feature included in the list2203from other features.

When the feature of the partial area included in the lesion area is changed, there may exist a case in which the diagnostic result is changed. That is, there may exist a case in which a feature capable of changing the diagnostic result by affecting the determination of the diagnostic result (e.g., benign or malignant) exists. Hereinafter, the feature capable of changing the diagnostic result is referred to as a “major feature.” In addition, a partial area indicating the major feature in the ultrasonic image is separately referred to as a second area.

For example, the processor420may acquire a contour area of a lesion and a contour of the lesion and acquire at least one feature among a shape of the lesion, an orientation of the lesion, a shape of the margin of the lesion, an echo with respect to a lesion area, and a posterior feature of the lesion area on the basis of the acquired the contour of the lesion.

Here, there may exist a case in which a section in which the contour of the lesion is unclear exists. For example, there may exist a case in which the contour of the lesion is unclear in a partial area (e.g.,2230) within a lesion area2220. In this case, in the partial area2230, the processor420may acquire a first contour as the contour of the lesion or acquire a second contour as the contour of the lesion. In this case, the shape of the lesion may be determined as being oval on the basis of the first contour, and the shape of the lesion may be determined as being irregular on the basis of the second contour. That is, when the contour of the lesion in the partial area (e.g.,2230) is changed, and when a feature that is the basis of the diagnostic result is changed, the processor420may control a diagnostic image2201, in which the partial area (e.g.,2230) is displayed, to be displayed.

In addition, the processor420may generate a diagnostic image by displaying a portion in which the contour of the lesion is unclear in the ultrasonic image. Here, the “portion in which the contour of the lesion is unclear” may be a portion in which accuracy of the extracted contour has a value that is less than or equal to a limit value. Specifically, the processor420may display the contour extracted from the portion in which the contour of the lesion is an unclear area (e.g.,2230) to be distinguished from a contour of a portion which is not unclear (e.g., a portion other than2230), thereby generating and displaying the diagnostic image2201.

Referring toFIG.22, the processor420may extract the lesion area2220having the illustrated contour. Here, a portion in which the extracted contour is unclear may be displayed as the partial area2230. Then, as shown inFIG.22, the processor420may control the diagnostic image2201, in which the partial area2230is displayed, to be displayed.

Specifically, as the analysis result of the lesion area2220having the illustrated contour, the processor420may determine a shape feature appearing in the partial area2230as being irregular. However, when the shape feature appearing in the partial area2230is changed to oval, there may exist a case in which the diagnostic result2250is changed from malignant to benign. Specifically, when the contour in the partial area2230is changed, the shape feature may be changed to oval, and when the shape feature is changed to the oval, the diagnostic result2250may be changed from malignant to benign. In this case, the shape feature may be the above-described major feature.

In the disclosed embodiment, the processor420may add an indication (e.g., a mark2275) for distinguishing the major feature appearing in the above-described unclear area (e.g.,2230) from other features in the list2202, thereby controlling the list2202to be generated and displayed. In addition, in an item corresponding to the major feature in the list2202, the processor420may add an indication (e.g., a mark2272) for distinguishing a currently acquired shape feature2271from other features (e.g., oval).

In addition, in the disclosed embodiment, when the contour of the lesion area extracted by the processor420includes the unclear portion (e.g., a section in which the accuracy of the contour is less than or equal to a limit value) (e.g.,2230), the processor420may control to display an icon2232for performing manual correction or contour re-extraction of an area corresponding to the partial area2230. For example, the icon2232may be expressed as a “c” mark indicating correction. Here, the manual correction may mean an operation in which a user manually corrects an unclear contour portion. Specifically, when a user input for selecting the icon2232is received, the processor420may control to output a user interface screen including an edit window (not shown) for correcting a contour within the partial area2230. As another example, when the icon2232is an icon for re-extraction of the contour, and when a user input for selecting the icon2232is received, the processor420may perform an automatic diagnostic operation, such as a CAD operation, to re-extract the contour and generate and display a diagnostic image including a lesion area formed by the re-extracted contour. In addition, the icon2232may be displayed in the partial area2230, which is an area requiring contour correction, or displayed at a position adjacent to the partial area2230.

In addition, in the disclosed embodiment, when a feature (i.e., a major feature) appearing in at least a partial area (e.g.,2230) included in the lesion area is changed and thus the diagnostic result is changed, the processor420may display the at least a partial area (e.g.,2230) in an ultrasonic image, thereby generating the diagnostic image2230. Hereinafter, for convenience of description, the above-described “at least a partial area (e.g.,2230)” will be referred to as a “second area.” In addition, the processor420may allow the user interface screen2200to include at least one among a symbol, a mark, and information which indicate that the diagnostic result may be changed. Specifically, the processor420may control the at least one among a symbol, a mark, and information, which indicate that the diagnostic result is changed when a feature appearing in the second area2230is changed, to be displayed in at least one among the diagnostic image2201, the diagnostic result2250, and the list2202.

Referring toFIG.22as an example, the processor420may detect the partial area2230that is the basis of determining a shape feature of the lesion area2220. As the analysis result of the lesion area2220, the processor420may determine the shape feature appearing in the partial area2230as being irregular. However, when the shape feature appearing in the partial area2230is changed to oval, there may exist a case in which the diagnostic result2250is changed from malignant to benign. That is, the shape feature may be the major feature.

In this case, the processor420may display an indication (e.g., the mark2275) in the list2202for distinguishing the major feature from other features in the list2202. In addition, in an item corresponding to the major feature in the list2202, the processor420may add the indication (e.g., the mark2272) for distinguishing a currently determined shape feature2271from other features (e.g., oval).

In addition, in displaying the diagnostic result2250, information indicating that the diagnostic result may be changed may be displayed.

In addition, when a plurality of major features are present among a plurality of features that are the basis of diagnosing a lesion, the processor420may display a plurality of second areas in the diagnostic image so as to distinguish the plurality of major features from each other. For example, when major features are a shape feature and a margin feature, the processor420may generate and display a diagnostic image in which a second area corresponding to the shape feature is displayed to be distinguished from a second area corresponding to the margin feature.

In addition, in determining whether the major feature exists, the processor420may consider whether a specific determination of a predetermined feature has accuracy that is greater than or equal to a limit value. Specifically, in the shape feature, an example in which, when the shape feature is determined as being oval, the accuracy is 70% and the limit value is 90% will be described. When the accuracy of determining that the shape feature is oval is 90%, since the accuracy has a value that is greater than and equal to the limit value, the processor420may not determine the shape feature as the major feature.

However, when the accuracy of determining that the shape feature is oval is 70%, since the accuracy has a value that is less than the limit value, the processor420may determine the shape feature as the major feature. Accordingly, when a predetermined feature determined as the major feature exists, the processor420may control the user interface screen2200to include at least one among the second area (e.g.,2230) indicating the major feature, an item (e.g.,2270) including detailed information on the major feature, and an indication indicating the presence of the major feature (e.g., the mark (2231or2275)).

FIG.23is a diagram illustrating another example of the user interface screen provided in the embodiment.

FIG.24is a diagram illustrating another example of the user interface screen provided in the embodiment.

InFIGS.23and24, the lesion area (e.g.,2220) included in the diagnostic image (e.g.,2201) is simplified and schematically illustrated. In addition, inFIGS.23and24, components that are the same as inFIG.22are illustrated using the same reference numerals.

Referring toFIG.23, when compared with the user interface screen2200, a user interface screen2300may further include a sub screen2302including detailed information2380corresponding to the major feature.

Here, the detailed information2380may include information indicating determination accuracy on the major feature (or a diagnosis probability). Specifically, an example in which a diagnosis probability of a shape feature to be determined as being oval is 20% (or 0.2), a diagnosis probability of a shape feature to be determined as being round is 70% (or 0.2), and a diagnosis probability of a shape feature to be determined as being irregular is 10% (or 0.2) will be described. Then, the processor420may control the information2380indicating the determination accuracy on the major feature to be generated and displayed on the user interface screen2300. Specifically, the information2380may include the diagnosis probability to be determined as being oval2381, the diagnosis probability to be determined as being round2382, and the diagnosis probability to be determined as being irregular2383. In addition, the diagnosis probability to be determined as being oval2381, the diagnosis probability to be determined as being round2382, and the diagnosis probability to be determined as being irregular2383may be displayed in the order of increasing probability values.

Referring toFIG.24, information2480included in a user interface screen2400may correspond to the information2380shown inFIG.23. Here, the information2480includes the same content as the information2380and, alternatively, may be expressed in a different format. For example, the diagnosis probability to be determined as being oval2381, the diagnosis probability to be determined as being round2382, and the diagnosis probability to be determined as being irregular2383may be expressed as a graph form2485in the user interface2400shown inFIG.24.

In the disclosed embodiment, when the ultrasound diagnosis device400outputs an ultrasonic image including a lesion area, in response to a user input in which one position or a predetermined partial area of the lesion area is selected, the processor420may control to generate or display a diagnostic image in which information on a feature of a lesion appearing in the selected position or the partial area or information that is the basis of a diagnostic result is displayed to overlap the ultrasonic image.

FIG.25is a diagram illustrating a diagnostic image output in response to a user input.

FIG.26is another diagram illustrating the diagnostic image output in response to the user input.

FIG.27is another diagram illustrating the diagnostic image output in response to the user input.

InFIGS.25to27, the same components are illustrated using the same reference numerals. In addition, an arrow marker (e.g.,2530) shown inFIGS.25to27indicates a position or a partial region, which is selected or pointed at through a user input. In addition, inFIGS.25to27, an ultrasonic image (e.g.,2501) includes a lesion area2510. In addition, inFIGS.25to27, the ultrasonic image2501, an ultrasonic image2601, and an ultrasonic image2701may correspond to each other identically.

Referring toFIG.25, the ultrasound diagnosis device400may display the ultrasonic image2501including the lesion area2510. In addition, the user may select a position or an area, at which the user specifically want to look, in the ultrasonic image2501. Here, a selection operation may be performed by manipulating (e.g., clicking or double clicking) a selection part such as a cursor or a pointer, which is included in the user interface460, to select a predetermined position or area. In addition, when a cursor or a pointer for selection is located on the ultrasonic image2501, the processor420may recognize that a corresponding position is selected. InFIGS.25to27, a location of the selection part for selection is illustrated as an arrow (e.g.,2530,2630, or2730).

Referring toFIG.25, when a predetermined position P1on the lesion area2510included in the ultrasonic image2501is selected, the processor420may control to generate and display a diagnostic image2502in which at least one of a partial area2550corresponding to the position P1selected in response to a selection input and information2551on a feature appearing in the partial area is displayed. For example, when the partial area2550corresponding to the selected position P1is an area indicating a shape feature among a plurality of features deriving a diagnostic result, the information2551such as “Shape: Oval” may be displayed in the diagnostic image2502.

Referring toFIG.26, when a predetermined position P2on the lesion area2510included in the ultrasonic image2601is selected, the processor420may control to generate and display a diagnostic image2602in which at least one of a partial area2650corresponding to the position P2selected in response to a selection input and information2651on a feature appearing in the partial area is displayed. For example, when the partial area2550corresponding to the selected position P2is an area indicating a posterior feature among a plurality of features deriving a diagnostic result, the information2651such as “No Posterior” may be displayed in the diagnostic image2602.

Referring toFIG.27, when a predetermined position P3on the lesion area2510included in the ultrasonic image2701is selected, the processor420may control to generate and display a diagnostic image2702in which at least one of a partial area2750corresponding to the position P3selected in response to a selection input and information2751on a feature appearing in the partial area is displayed. For example, when the partial area2750corresponding to the selected position P3is an area indicating an echo feature among a plurality of features deriving a diagnostic result, the information2751such as “Echo Pattern: Hypoechoic” may be displayed in the diagnostic image2702.

In addition, even when the user input is not present, the processor420may move a contour line of the lesion area2510included in the ultrasonic image (e.g.,2501) in a clockwise orientation or a counterclockwise orientation and control to generate and display the diagnostic image (e.g.,2502) in which at least one of an area corresponding to a position at which the contour line is moved and information (e.g.,2551) on a feature appearing in the area is displayed.

In addition, the processor420may divide the contour line of the lesion area2510included in the ultrasonic image (e.g.,2501) into a plurality of sections and control to generate and display a diagnostic image in which at least one of an area corresponding to the divided section and information on a feature appearing in the area. In addition, the plurality of sections may be displayed to be distinguished from each other using at least one among different colors, marks, symbols, transparency, echoes, types of dotted lines, and thicknesses of the dotted lines.

FIG.28is a diagram illustrating a user interface screen including a plurality of diagnostic images.

In the disclosed embodiment, in order to identify the lesion area, the processor420may extract the contour of the lesion using at least one of a CAD technology and an AI technology. In addition, when a plurality of extracted contours of the lesion are present, the processor420may control to generate and display a user interface screen including a plurality of ultrasonic images in which the plurality of extracted contours of the lesion are displayed.

In addition, when a plurality of extracted contours of the lesion are present and each of the plurality of extracted contours corresponds to a different diagnostic result, the processor420may control to generate and display a user interface screen including a plurality of ultrasonic images in which the plurality of extracted contours of the lesion are displayed.

Referring toFIG.28, a user interface screen2800may include a plurality of ultrasonic images2810,2820,2830,2840,2850, and2860. Here, each of the plurality of ultrasonic images2810,2820,2830,2840,2850, and2860included in the user interface screen2800may be an ultrasonic image in which a different contour is displayed.

In addition, each of the plurality of ultrasonic images2810,2820,2830,2840,2850, and2860may be disposed in the order of accuracy of an extracted contour. For example, the ultrasonic image2810in which a contour2801having the highest contour accuracy is displayed may be disposed at a first position, and the ultrasonic image2820in which a contour2821having second highest contour accuracy is displayed may be disposed at a second position. In addition, the ultrasonic image2830in which a contour2831having third highest contour accuracy is displayed may be disposed at a third position, and the ultrasonic image2840in which a contour2841having fourth highest contour accuracy is displayed may be disposed at a fourth position.

In addition, in each of the plurality of ultrasonic images2820,2830,2840,2850, and2860, the contour2801having the highest contour accuracy may be illustrated as a comparison target.

In addition, a diagnostic result (not shown inFIG.28) derived on the basis of the extracted contour may be displayed in each of the plurality of ultrasonic images2810,2820,2830,2840,2850, and2860.

In addition, when each of the plurality of ultrasonic images2810,2820,2830,2840,2850, and2860is selected on the user interface screen2800, a diagnostic image including diagnostic information corresponding to the diagnostic result may be enlarged and displayed.

FIG.29is a diagram for describing a neural network used for acquiring an analysis result of an ultrasonic image.

In the disclosed embodiment, the processor420may analyze a lesion using at least one of a CAD technology and an AI technology and acquire at least one among a lesion area, a diagnostic result, and diagnostic information corresponding to the diagnostic result.

Hereinafter, an operation of acquiring at least one among the lesion area, the diagnostic result, and the diagnostic information corresponding to the diagnostic result using an AI technology will be described.

The AI technology is a technology for obtaining a desired result by performing an operation through a neural network to analyze an input image or input data. Here, the neural network may train training data (e.g., a plurality of different ultrasonic images) to optimize and set weight values in the neural network. Then, the neural network learns the input data by itself through a neural network having the optimized weight value, thereby outputting a desired result.

Specifically, the neural network may be a DNN. In addition, an operation of the DNN may include an operation of a CNN. Specifically, a data recognition model may be implemented through an exemplified neural network, and the implemented data recognition model may be trained using training data. In addition, data to be input, for example, an ultrasonic image including a lesion area, may be analyzed or classified using the learned data recognition model, and a specific area (e.g., a contour area of the lesion) or a specific area image (e.g., an image corresponding to a partial area that is the basis of deriving a diagnostic result, as in a first area), which is included in the ultrasonic image, may be analyzed and/or classified.

In the embodiment of the present disclosure, the processor420may perform an operation through a neural network to acquire at least one among the lesion area, the diagnostic result, and the diagnostic information corresponding to the diagnostic result from the ultrasonic image.

Alternatively, the above-described operation through the neural network may be performed by the image processing unit450.

Alternatively, the above-described operation through the neural network may performed through an external server (not shown), and the operation result through the neural network (e.g., at least one among the lesion area, the diagnostic result, and the diagnostic information corresponding to the diagnostic result) may be received through the communication unit470.

The processor420may perform an operation through a neural network including an input layer, a hidden layer, and an output layer, for example, a DNN2920. InFIG.9, a case in which a hidden layer included in a neural network is formed as a DNN formed of multiple stages is illustrated as an example.

Referring toFIG.29, the DNN2920includes an input layer2930, a hidden layer2940, and an output layer2950. InFIG.29, the DNN2920which analyzes information included in an ultrasonic image that is input data and outputs desired output data2970is illustrated as an example. Here, the output data2970may include at least one among a lesion area, a diagnostic result, and diagnostic information corresponding to the diagnostic result.

A plurality of layers forming the DNN2920may include a plurality of nodes (e.g.,2931) for receiving data. In addition, as shown in the drawing, two adjacent layers are connected by a plurality of edges (e.g.,2936). Each node has a corresponding weight value. Accordingly, the DNN2920may acquire output data by calculating an input signal and a weight value, for example, on the basis of a value obtained by performing a convolution operation.

Here, the DNN2920may be formed as a CNN neural network which performs a CNN operation.

Referring to the example shown inFIG.29, the input layer2930receives an ultrasonic image2910including a lesion area2902.

InFIG.29, a case in which the hidden layer2940is formed of three-stage layers is illustrated as an example. A depth of the hidden layer2940may be varied according to an nthorder specification and/or design specification of a used neural network.

Referring toFIG.29, the DNN2920includes a first layer Layer 1 formed between the input layer2930and a first hidden layer HIDDEN LAYER1, a second layer Layer 2 formed between the first hidden layer HIDDEN LAYER1 and a second hidden layer HIDDEN LAYER2, a third layer Layer 3 formed between the second hidden layer HIDDEN LAYER2 and a third hidden layer HIDDEN LAYER3, and a fourth layer Layer 4 formed between the third hidden layer HIDDEN LAYER3 and the output layer2970.

A plurality of nodes included in the input layer2930of the DNN2920receive a plurality of pieces of data corresponding to the ultrasonic image2910. Here, the plurality of pieces of data may be a plurality of partial images generated by performing a filter process to divide the ultrasonic image2910.

In addition, the output layer2950may output the output data2970acquired as the analysis result of the ultrasonic image2910through operations in the plurality of layers included in the hidden layer2940.

Specifically, when the DNN2920is formed of a CNN and when a correlation between information included in an image is local, the CNN may introduce a concept of a filter illuminating only a specific area and perform a convolution operation on pieces of information in the filter, thereby precisely extracting information on a feature of an image in the filter.

Specifically, a convolution layer and a pooling layer are alternately disposed in the hidden layer2940existing in the CNN-based neural network2920, and a depth of a filter of each layer is increased from left to right. In addition, the final stage of the CNN-based neural network2920may be formed as a fully connected layer. Here, the convolution layer is a layer of data generated according to a convolution operation, and the pooling layer is a layer for reducing the number of pieces of data or size of data through an operation such as subsampling or pooling. Pieces of data (e.g., a feature map) indicating a feature of an input image is generated by passing through the convolution layer and the pooling layer. Specifically, image features of the ultrasonic image2910are generated through the operation of the hidden layer2940, and a lesion area, a diagnostic result with respect to the lesion, and diagnostic information corresponding to the diagnostic result may be acquired on the basis of the image features.

In addition, when the pieces of data generated by passing through the convolutional layer and the pooling layer are processed through the hidden layer formed as a fully connected layer, the desired output data2970may be output.

In addition, in order to increase accuracy of the output data output through the DNN2920, training may be performed in an orientation from the output layer2950to the input layer2930, and in order to increase accuracy of the output data, weight values of nodes (e.g.,2931) forming the DNN2920may be corrected. Therefore, before the ultrasonic image2910is input, the DNN2920may learn a plurality of ultrasonic images including a plurality of different lesions to correct a weight value of each node in an orientation of a lesion area included in the ultrasonic image or in an orientation in which a diagnostic result is accurately detected.

In addition, the DNN2920may perform an operation through the neural network to acquire a criterion for detecting the diagnostic result. For example, criteria indicating features of a malignant lesion or criteria indicating feature values of a benign lesion may be acquired.

FIG.30is a flowchart illustrating a method of displaying an ultrasonic image according to an embodiment. A method3000of displaying an ultrasonic image according to the disclosed embodiment may be performed through the ultrasound diagnosis device100,400, or500according to the disclosed embodiments described with reference toFIGS.1to29. Therefore, each operation of the method3000of displaying an ultrasonic image may be performed through each configuration of the ultrasound diagnosis device100,400, or500, and the method3000of displaying an ultrasonic image may include the same configurational features as in the above-described ultrasound diagnosis device100,400, or500. That is,FIG.30may be a diagram illustrating operations performed by the ultrasound diagnosis device100,400or500according to the disclosed embodiments. Therefore, in describing the method3000of displaying an ultrasonic image, descriptions overlapping those ofFIGS.1to29will be omitted herein.

The method3000of displaying an ultrasonic image identifies a lesion area included in an ultrasonic image (S3010). Operation S3010may be performed by the processor420.

The lesion area identified in operation S3010is diagnosed to acquire a diagnostic result (S3020). Operation S3020may be performed by the processor420.

Then, in the lesion area, a first area, which is at least one area that is the basis of diagnosing a lesion, is displayed in the lesion area of the ultrasonic image to generate a diagnostic image (S3030). Operation S3030may be performed by the processor420.

Then, a user interface screen including the diagnostic image and the diagnostic result is displayed (S3040). Operation S3040may be performed by the processor420.

FIG.31is a diagram illustrating an actual implementation example of the user interface screen according to the embodiment.

FIG.32is another diagram illustrating an actual implementation example of the user interface screen according to the embodiment.

Specifically, inFIGS.31and32, an actual user interface screen output from the ultrasound diagnosis device implemented according to the disclosed embodiments is illustrated.

Referring toFIG.31, a user interface screen3100may include a diagnostic image3110generated by marking an extracted contour area in an ultrasonic image, a diagnostic result3120, and a list3130indicating features of a lesion area. Here, the diagnostic image3110, the diagnostic result3120, and the list3130indicating the features of the lesion area identically correspond to the diagnostic image1310, the diagnostic result1330, and the list1302indicating the features of the lesion area shown inFIG.13, respectively, and thus detailed descriptions thereof will be omitted herein.

In the disclosed embodiment, due to selection of at least one item in the list3130or a setting of the processor420, the first area may be displayed in the diagnostic image3110.

Referring toFIG.32, when compared to the user interface screen3200, a user interface screen3200may further include ultrasound scan information3240with respect to a patient or an object.

Meanwhile, the disclosed embodiments may be implemented in the form of a computer-readable recording medium storing commands executable by a computer and data. The command may be stored in the form of a program code, and when the command is executed by a processor, a predetermined program module may be generated to perform a predetermined operation. In addition, when the command is executed by a processor, the command may perform predetermined operations of the disclosed embodiments.

The computer-readable medium may include program instructions, data files, data structures, and the like in alone or a combination thereof. The program command recorded in the computer-readable medium may be specially designed and configured for the embodiment or may be available to those skilled in the computer software. Examples of the computer-readable recording media include magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical recording media such as a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD), a magneto-optical medium such as a floptical disk, and hardware devices specifically configured to store and execute program commands, such as a ROM, a RAM, a flash memory, and the like. Examples of the program commands include machine language codes generated by a compiler as well as high-level language codes which are executable by a computer using an interpreter or the like.

In addition, the method of displaying an ultrasonic image according to the above-described embodiment may be implemented as a computer program product including a recording medium in which a program is stored, wherein the program performs an operation of acquiring a sentence composed of multiple languages; and an operation of acquiring vector values corresponding to words included in the multilingual sentence using a multilingual translation model, converting the acquired vector values into vector values corresponding to a target language, and acquiring a sentence composed of the target language on the basis of the converted vector values.