Patent ID: 12186132

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately.

FIG.1is a schematic diagram of an ultrasound imaging system100, according to aspects of the present disclosure. The system100is used for scanning an area or volume of a patient's body. The system100includes an ultrasound imaging probe110in communication with a host130over a communication interface or link120. The probe110includes a transducer array112, a beamformer114, a processor circuit116, and a communication interface118. The host130includes a display132, a processor circuit134, and a communication interface136.

In an exemplary embodiment, the probe110is an external ultrasound imaging device including a housing configured for handheld operation by a user. The transducer array112can be configured to obtain ultrasound data while the user grasps the housing of the probe110such that the transducer array112is positioned adjacent to and/or in contact with a patient's skin. The probe110is configured to obtain ultrasound data of anatomy within the patient's body while the probe110is positioned outside of the patient's body. In some embodiment, the probe110can be an external ultrasound probe suitable for abdominal examination, for example, for diagnosing appendicitis or intussusception.

The transducer array112emits ultrasound signals towards an anatomical object105of a patient and receives echo signals reflected from the object105back to the transducer array112. The ultrasound transducer array112can include any suitable number of acoustic elements, including one or more acoustic elements and/or plurality of acoustic elements. In some instances, the transducer array112includes a single acoustic element. In some instances, the transducer array112may include an array of acoustic elements with any number of acoustic elements in any suitable configuration. For example, the transducer array112can include between 1 acoustic element and 10000 acoustic elements, including values such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements, 1000 acoustic elements, 3000 acoustic elements, 8000 acoustic elements, and/or other values both larger and smaller. In some instances, the transducer array112may include an array of acoustic elements with any number of acoustic elements in any suitable configuration, such as a linear array, a planar array, a curved array, a curvilinear array, a circumferential array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x dimensional array (e.g., a 1.5D array), or a two-dimensional (2D) array. The array of acoustic elements (e.g., one or more rows, one or more columns, and/or one or more orientations) that can be uniformly or independently controlled and activated. The transducer array112can be configured to obtain one-dimensional, two-dimensional, and/or three-dimensional images of patient anatomy. In some embodiments, the transducer array112may include a piezoelectric micromachined ultrasound transducer (PMUT), capacitive micromachined ultrasonic transducer (CMUT), single crystal, lead zirconate titanate (PZT), PZT composite, other suitable transducer types, and/or combinations thereof.

The object105may include any anatomy, such as blood vessels, nerve fibers, airways, mitral leaflets, cardiac structure, abdominal tissue structure, appendix, large intestine (or colon), small intestine, kidney, and/or liver of a patient that is suitable for ultrasound imaging examination. In some aspects, the object105may include at least a portion of a patient's large intestine, small intestine, cecum pouch, appendix, terminal ileum, liver, epigastrium, and/or psoas muscle. The present disclosure can be implemented in the context of any number of anatomical locations and tissue types, including without limitation, organs including the liver, heart, kidneys, gall bladder, pancreas, lungs; ducts; intestines; nervous system structures including the brain, dural sac, spinal cord and peripheral nerves; the urinary tract; as well as valves within the blood vessels, blood, chambers or other parts of the heart, abdominal organs, and/or other systems of the body. In some embodiments, the object105may include malignancies such as tumors, cysts, lesions, hemorrhages, or blood pools within any part of human anatomy. The anatomy may be a blood vessel, as an artery or a vein of a patient's vascular system, including cardiac vasculature, peripheral vasculature, neural vasculature, renal vasculature, and/or any other suitable lumen inside the body. In addition to natural structures, the present disclosure can be implemented in the context of man-made structures such as, but without limitation, heart valves, stents, shunts, filters, implants and other devices.

The beamformer114is coupled to the transducer array112. The beamformer114controls the transducer array112, for example, for transmission of the ultrasound signals and reception of the ultrasound echo signals. The beamformer114provides image signals to the processor circuit116based on the response of the received ultrasound echo signals. The beamformer114may include multiple stages of beamforming. The beamforming can reduce the number of signal lines for coupling to the processor circuit116. In some embodiments, the transducer array112in combination with the beamformer114may be referred to as an ultrasound imaging component.

The processor circuit116is coupled to the beamformer114. The processor circuit116may include a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor circuit134may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor circuit116is configured to process the beamformed image signals. For example, the processor circuit116may perform filtering and/or quadrature demodulation to condition the image signals. The processor circuit116and/or134can be configured to control the array112to obtain ultrasound data associated with the object105.

The communication interface118is coupled to the processor circuit116. The communication interface118may include one or more transmitters, one or more receivers, one or more transceivers, and/or circuitry for transmitting and/or receiving communication signals. The communication interface118can include hardware components and/or software components implementing a particular communication protocol suitable for transporting signals over the communication link120to the host130. The communication interface118can be referred to as a communication device or a communication interface module.

The communication link120may be any suitable communication link. For example, the communication link120may be a wired link, such as a universal serial bus (USB) link or an Ethernet link. Alternatively, the communication link120may be a wireless link, such as an ultra-wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

At the host130, the communication interface136may receive the image signals. The communication interface136may be substantially similar to the communication interface118. The host130may be any suitable computing and display device, such as a workstation, a personal computer (PC), a laptop, a tablet, or a mobile phone.

The processor circuit134is coupled to the communication interface136. The processor circuit134may be implemented as a combination of software components and hardware components. The processor circuit134may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, a FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processor circuit134may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The processor circuit134can be configured to generate image data from the image signals received from the probe110. The processor circuit134can apply advanced signal processing and/or image processing techniques to the image signals. In some embodiments, the processor circuit134can form three-dimensional (3D) volume image from the image data. In some embodiments, the processor circuit134can perform real-time processing on the image data to provide a streaming video of ultrasound images of the object105.

The display132is coupled to the processor circuit134. The display132may be a monitor or any suitable display. The display132is configured to display the ultrasound images, image videos, and/or any imaging information of the object105.

The system100may be used to assist a sonographer in performing an ultrasound scan, for example, at a point-of-care setting. For instance, the host130may be a mobile device, such as a tablet or a mobile phone. In some aspects, the sonographer may place the probe110on a patient to begin an ultrasound scan for a certain clinical evaluation, for example, for appendicitis or intussusception. The sonographer may follow a certain scanning topography or scanning trajectory commonly used or recommended for the clinical evaluation. The scanning topography or scanning trajectory may include a series of anatomical landmarks that may be relevant to the evaluation or may lead to a target anatomical site for the evaluation.

According to embodiments of the present disclosure, the system100may acquire an ultrasound image of the patient's anatomy at the initial position of the probe110, determine a location within the patient's anatomy with respect to the scanning trajectory based on the image and/or an anatomical objected detected from the image, and determine a steering configuration for steering the probe110towards a next landmark along the scanning trajectory based on the determined location and/or the detected anatomical object. The system100may display graphical or visual indicators on the display132based on the steering configuration to guide the sonographer in steering the probe110towards the next landmark. The sonographer may steer the probe110based on the graphical or visual indicators. The system100may provide automated systematic scan guidance to the sonographer in real-time by repeatedly acquiring ultrasound images and determining steering configurations for steering the probe110towards a landmark along the scanning trajectory as the sonographer moves the probe110. Additionally, the system100can automatically determine adjustments for ultrasound signal settings, such as signal gain and/or an imaging depth of field, and/or beam steering based on the acquired images. For instance, the system100may include a probe controller150configured to receive instruction from the host130and configure the transducer array112at the probe110. Further, the system100can automatically capture or save one or more of the acquired images in the system100(e.g., at the memory138) that are representative of a normal clinical condition for the clinical evaluation or an abnormal condition for the clinical evaluation so that the sonographer may review the captured or saved images after the scan.

In some aspects, the processor circuit134may implement one or more deep learning-based prediction networks trained to perform classification and/or prediction based on input ultrasound images. The classification and/or prediction may include classifying or identifying anatomical landmarks along a scanning trajectory for a certain clinical evaluation, predicting a steering configuration (e.g., including rotations and/or translations) for steering the probe110along the scanning trajectory and/or towards a certain target anatomical site, predicting a beam steering angle for configuring the transducer array112to reach a certain anatomical landmark or target site, predicting the presence or absence of a certain clinical condition, and/or predicting a ultrasound signal setting based on the quality of the image. Mechanisms for providing systematic scan assistance are described in greater detail herein.

In some aspects, the system100can be used for collecting ultrasound images to form training data set for deep learning network training. For example, the host130may include a memory138, which may be any suitable storage device, such as a cache memory (e.g., a cache memory of the processor circuit134), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, solid state drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. The memory138can be configured to store an image data set140to train a deep learning network in providing automatic scan assistance, automatic image capture, automatic ultrasound signal setting adjustments, and/or automatic beam steering. Mechanisms for training the prediction or deep learning networks are described in greater detail herein.

In clinical practices, when a patient is sent to an emergency room with an abdominal pain, an emergency physician may perform an abdominal scan to determine the cause of the abdominal pain. Two common causes of abdominal pain are appendicitis and intussusception. Appendicitis is the inflammation of the appendix (shown inFIG.2). Intussusception is a medical condition in which one segment of the intestine telescopes inside of another segment of intestine, causing an intestinal obstruction or blockage (shown inFIG.3).

FIG.2is a schematic diagram illustrating an appendix210in an abdominal structure200, according to aspects of the present disclosure. The system100may be used to perform a scan around a patient's abdominal area (e.g., the abdominal structure200) for evaluating an appendicitis condition. The appendix210is a closed tube of tissue attached to the large intestine220(also referred to as a colon) in the lower right abdomen. Inflammation occurs when the appendix210becomes infected or blocked. For instance, when the appendix210is inflamed, the appendix210may have a diameter larger than about 6 millimeters (mm). The sonographer may also use other characteristics to diagnose an appendicitis condition, for example, when the appendix210is fluid-filled. However, some of the structures surrounding or near the appendix210, such as the appendix epiploicae230, theTaenia coli240, the cecum250, the ileocolic artery260, the ileum270(a final section of the small intestine), the mesoappendix280, and/or the appendicular artery290, can cause challenges in locating the appendix210during the scan.

FIG.3is a schematic diagram illustrating an intussusception condition300, according to aspects of the present disclosure. The system100may be used to perform an abdominal scan for diagnosing the intussusception condition300. Intussusception can occur anywhere in the gastrointestinal tract. One of the most common form of intussusception may be ileocolic intussusception, which occurs at the junction of the small and large intestines.FIG.3illustrates a portion of a patient's gastrointestinal tract including a large intestine310and a small intestine320. A normal portion of the large intestine310is zoomed-in and shown as302, where the small intestine320does not cause any obstruction or blockage to the large intestine310. A portion of the large intestine310including an intussusception330is zoomed-in and shown as304. At the site of the intussusception330, a segment of the small intestine320folds into the large intestine310, causing blockages.

For abdominal ultrasound scans, there are several abdominal topographies that physicians may use. One of the abdominal topographies that is preferred by many physicians is shown inFIG.4A.FIG.4Ais a schematic diagram of an abdominal topography400, according to aspects of the present disclosure. The system100can be used to perform a systematic abdominal scan according to the topography400. As shown, the topography400divides a patient's abdominal area402by an axial line and a transverse line through the umbilicus into a right upper quadrant (RUQ)410, a right lower quadrant (RLU)420, a left upper quadrant (LUQ)430, and a left lower quadrant (440). Scanning for appendicitis and intussusception requires systematic scanning and covering the topography400for clinical condition evaluation. However, an abdominal scan for diagnosing appendicitis may use a different trajectory over the topography400than an abdominal scan for diagnosing intussusception as shown and described below inFIG.4BandFIG.4C.

FIG.4Bis a schematic diagram illustrating a scanning trajectory450for an appendicitis assessment, according to aspects of the present disclosure. The system100may be used to perform an ultrasound scan according to the scanning trajectory450. An appendicitis assessment may include a systematic scanning of the RLQ (e.g., the RLQ420) of a patient. The scanning trajectory450covers the RLQ of the patient as shown by the arrows451-454. For instance, a sonographer may place a probe459(e.g., the probe110) transversely at an initial location in the RLQ near the umbilicus, sweep left as shown by the arrow451, followed by sweeping down as shown by the arrow452, sweeping right as shown by the arrow453, and sweeping up and down as shown by the arrow454to locate the appendix (e.g., the appendix210) for an appendicitis assessment.

FIG.4Cis a schematic diagram illustrating a scanning trajectory460for an intussusception assessment, according to aspects of the present disclosure. The system100may be used to perform an ultrasound scan according to the scanning trajectory460. An intussusception assessment may include a systematic scanning of the entire abdominal topography (e.g., the topography400), for example, in a lawn mow pattern. The scanning trajectory460are shown by the arrows461-463. For instance, a sonographer may place a probe469(e.g., the probe110) transversely at an initial location in the RLQ (e.g., the RLQ420) of the patient, sweep up as shown by the arrow461, followed by sweeping right as shown by the arrow462with the probe459placed longitudinally, and followed by sweeping down as shown by the arrow463with the probe459place transversely for an intussusception assessment.

While computerized tomography (CT) scan may be used for abdominal assessment, ultrasound is a preferred imaging modality due to the easy availability and lack of radiation. Additionally, the sensitivity and specificity of ultrasound is high, thus the use of ultrasound has become a common clinical practice for scanning for appendicitis or intussusception. Further, POCUS enables the ease of use and the availability of ultrasound at point-of-care settings.

However, for novice users, it is often challenging to perform serial, rapid, focused assessments in emergency settings. During ultrasound scanning of appendicitis or intussusception, an older child can point to the location of the pain but for younger patients, it requires systematic scanning. Though novice users can choose an initial location and try to follow a systematic approach, they may encounter several challenges while attempting to follow the ideal abdominal trajectory (e.g., the trajectories450and460) to diagnose the condition. Some examples of challenges may include misidentification of the structure (e.g., misidentifying a terminal ileum or another component of the small bowel as the appendix), missed detection of the retrocecal (localized behind the cecum) appendicitis. The need to identify landmarks like ascending colon and/or wall of the colon and/or confirmation of sufficient scan coverage can also be challenging. Additionally, the identifying of features to support a certain diagnosis, for example, for appendicitis, such as measuring dilated appendix and determining whether the appendix has a diameter larger than 6 mm, searching for different clinical presentation of appendicolith (an appendicolith is composed of firm feces and some mineral deposits), identifying echogenic prominent peri-cecal fat, per-appendicular fluid collection, and/or searching for specific patterns (e.g., target like appearance appendix versus intussusception), can be time-consuming. Further, the use of ultrasound for diagnosis can be dependent on sonographer experiences.

Accordingly, the present disclosure provides techniques to automatically detect anatomical landmarks along a certain scanning trajectory or topography to provide systematic scan guidance to a sonographer. In some aspects, the present disclosure provides automatic probe steering guidance for scanning along a certain scanning trajectory relevant for a specific clinical examination, automatic capture of imaging frames that are representative of a normal clinical condition and/or an abnormal clinical condition to enable a quicker diagnosis, and/or automatic ultrasound signal setting adjustments for optimal imaging quality. In some aspects, the present disclosure may utilize a deep learning framework trained to provide automatic scanning guidance, auto-capture, and/or auto-signal-adjustments. Mechanisms for providing automatic scanning guidance, auto-capture, and/or auto-signal-adjustments are described in greater detail herein.

FIG.5is a schematic diagram of an automated ultrasound scanning assistance system500, according to aspects of the present disclosure. The system500may correspond to the system100, and may provide a detailed view of an implementation for providing ultrasound scanning assistance. At a high level, the system500automatically identifies anatomical landmarks from acquired images for a certain scanning trajectory (e.g., the trajectory450or460) related to a clinical assessment, translate the identified information into user actions to provide a sonographer with systematic scan guidance. The system500can provide visual guidance to steer a probe (e.g., the probe110) along the scanning trajectory and/or textual display of the automatically identified landmarks. Additionally, the system500automatically adjusts ultrasound signal gain settings (e.g., signal gain and/or signal depth) to provide an optimal imaging view. Further, the system500automatically captures and saves images frames that are relevant for a certain clinical assessment and tracks whether the scan includes a sufficient coverage for the clinical assessment.

The system500includes a probe510, an image acquisition component520, a data preprocessing component530, a prediction component540, a deep learning model repository580, an auto-capture component550, an auto-scan setting component560, and a visual guidance component570. The probe510may correspond to the probe110ofFIG.1and may be capable of 2D imaging or 3D imaging. The visual guidance component570may correspond to the display132ofFIG.1. The image acquisition component520, the data preprocessing component,530, the prediction component540, the auto-capture component550, and/or the auto-scan setting component560may include hardware and/or software components. The image acquisition component520may include interfaces for receiving image frames from the probe510. In some instances, the data preprocessing component,530, the prediction component540, the auto-capture component550, and/or the auto-scan setting component560are implemented by the processor circuit134. In some instances, the data preprocessing component,530, the prediction component540, the auto-capture component550, and/or the auto-scan setting component560may be implemented by additional dedicated hardware configured for performing corresponding functions. The deep learning model repository580may include one or more trained deep learning networks582saved in a memory (e.g., the memory138).

The probe510is configured to acquire ultrasound images512. The images512are received by the image acquisition component520. The images512may be brightness-mode (B-mode) images. The data preprocessing component530is configured to preprocess the imaging data532from the images512before passing the imaging data532to the prediction component540. The preprocessing may include resizing the images512, cropping a region of interest from the images512, adjusting imaging gain, and/or any suitable image preprocessing.

The prediction component540receives the preprocessed imaging data532. The imaging data532may be in the form of imaging frame. The prediction component540is configured to apply a trained deep learning network582stored in the deep learning model repository580to an input image512. The deep learning network582performs the tasks of anatomical object classification and/or detection and probe steering prediction. In this regard, the deep learning network582may detect anatomical landmarks for a certain scanning trajectory and/or identifying a location of the probe510with respect to the scanning trajectory and/or landmarks from each acquired image512. The deep learning network582may predict a steering configuration for steering the probe510to scan along the scanning trajectory based on the location of the probe510and/or the detected landmarks. The deep learning network582can be trained to identify landmarks for a certain scanning trajectory, for example, for an appendicitis evaluation or an intussusception evaluation, and infer a steering configuration based on identified landmarks. The detected landmarks served as the input to derive the steering actions. Some example anatomical landmarks along a scanning trajectory for appendicitis evaluation may include liver, ascending colon, cecum pouch, terminal ileum, appendix/appendicitis, and/or target signatures appendicitis. Some example anatomical landmarks along a scanning trajectory for intussusception evaluation may include psoas muscle, ascending colon, liver, gall bladder, epigastrium, descending colon, and target signatures of intussusception. The architecture of the deep learning network582is discussed in greater detail below inFIG.6and the training of the deep learning network is discussed in greater detail below inFIGS.7and8. Scanning trajectories for appendicitis and intussusception are discussed in greater detail below inFIGS.9and10.

The prediction component540outputs scan guidance information546predicted by the deep learning network582to the visual guidance component570. The scan guidance information546may include the probe steering configuration and/or the identified landmarks. The steering configuration may include translations and/or rotations of the probe510. The visual guidance component570is configured to translate the steering configuration into actions and display the actions as visual guidance to instruct the sonographer to steer the probe510. The actions may include changing the probe510position, orientation, and/or direction. The visual guidance component570may also output the identified landmarks in textual form or display an acquired image512with the identified landmarks overlaid on top of the acquired image512.

During a clinical examination, a sonographer may place the probe510on a patient at an initial location close to the scanning trajectory or somewhere along a starting position of the scanning trajectory. The probe510may acquire a first image512of the patient's anatomy. The deep learning network582may predict a steering configuration based on the first image512to steer the probe510towards a next landmark. The visual guidance component570may display steering instructions to guide the sonographer in steering the probe510. The process of steering the probe510, acquiring images, and predicting steering configurations based on the acquired images may continue until the scan covers the entire scanning trajectory. The clinical workflow and scan guidance are discussed in greater detail below inFIGS.9and10.

In some aspects, the probe510may include a 2D transducer array capable of 3D imaging. The deep learning network582is trained to infer an ultrasound beam angle547from an input image512for steering ultrasound beams of the transducer array towards a target site (e.g., the appendix210or an intussusception330) instead of instructing the sonographer to rotate the probe510. The system500may include an electronic auto-beam steering component590configured to steer the ultrasound beams at the probe510based on the angle547predicted by the deep learning network582. For instance, the electronic auto-beam steering component590can output a control signal592to adjust the beam angle at the probe510.

Additionally, the deep learning network582is trained to determine whether an input image512has a good imaging quality or a bad imaging quality. In some instances, the prediction component540outputs indication544of the imaging quality. The indication544may indicate a good imaging quality or a bad imaging quality. The auto-scan setting component560receives the feedback (e.g., the indication544) from the prediction component540and automatically determine an ultrasound signal setting adjustment for the probe510. For instance, the adjustment may include a signal gain, for example, by controlling gain stages at the frontend of the probe510. Additionally or alternatively, the adjustment may include an imaging depth of field adjustment, for example, by increasing or decreasing the ultrasound frequencies to decrease or increase the depth of view, respectively. The auto-scan setting component560outputs a control signal562to adjust the ultrasound signal setting at the probe510according to the indication544. In some other instances, the deep learning network582is trained to infer an ultrasound signal setting (e.g., a signal gain or an imaging depth of field adjustment) based on the identified image quality and the prediction component540can instruct the auto-scan setting component560to adjust the ultrasound signal setting on the probe510. While the auto-scan setting component560and the auto-beam steering component590are shown as separate components inFIG.5, in some examples, the auto-scan setting component560and the auto-beam steering component590can be a single controller configured to control ultrasound signal setting and/or beam angle at the probe510.

Further, the deep learning network582is trained to determine whether an input image512is relevant for a clinical diagnosis and output images542that are relevant for the diagnosis to the auto-capture component550. The images542may include images with anatomical features representative of a normal condition and/or an abnormal condition for the clinical diagnosis. The auto-capture component550is configured to save the still image542to a memory (e.g., the memory138) at the system500. The sonographer may refer to the auto-captured or auto-saved image542to determine a clinical condition. Thus, the auto-captured or auto-saved image542can reduce time for the sonographer to search for relevant imaging frames for the clinical diagnosis.

In some aspects, the auto-capture component550may determine whether a sufficient coverage is achieved from the scan for the clinical assessment based on the acquired images512. Alternatively, the deep learning network582is further trained to determine whether a sufficient coverage of the scanning trajectory had been scanned based on the captured images512. If the scan does not include a sufficient coverage of the scanning trajectory for the clinical assessment, the prediction component540can continue to apply the deep learning network582to provide scanning guidance to the sonographer until the scan covers a sufficient coverage of the scanning trajectory.

In some aspects, the prediction component540may apply different trained deep learning networks from the deep learning model repository580for different types of predictions (e.g., classification, object detection, image quality prediction, auto-signal setting prediction, abnormal clinical condition/normal clinical condition prediction). In some aspects, the prediction component540may apply a single deep learning network for the predictions discussed above. In some aspects, the prediction component540may apply different deep learning networks for different types of predictions. For instance, the deep learning model repository580may include one deep learning network trained for probe steering motion prediction and another deep learning network trained for image quality or signal setting prediction. In some aspects, the prediction component540may apply different deep learning networks for different types of assessments. For instance, the deep learning model repository580may include one deep learning network trained to assist scanning for appendicitis assessment and another deep learning network trained to assist scanning for intussusception.

FIG.6is a schematic diagram of a deep learning network configuration600, according to aspects of the present disclosure. The configuration600can be implemented by a deep learning network such as the deep learning network582. The configuration600includes a deep learning network610including one or more CNNs612. For simplicity of illustration and discussion,FIG.6illustrates one CNN612. However, the embodiments can be scaled to include any suitable number of CNNs612(e.g., about 2, 3 or more). The configuration600can be trained for automated anatomical landmark detection, inference of probe steering configuration, image quality detection, and/or clinical condition detection as described in greater detail below.

The CNN612may include a set of N convolutional layers620followed by a set of K fully connected layers630, where N and K may be any positive integers. The convolutional layers620are shown as620(1)to620(N). The fully connected layers630are shown as630(1)to630(K). Each convolutional layer620may include a set of filters622configured to extract features from an input602(e.g., images512). The values N and K and the size of the filters622may vary depending on the embodiments. In some instances, the convolutional layers620(1)to620(N)and the fully connected layers630(1)to630(K-1)may utilize a leaky rectified non-linear (ReLU) activation function and/or batch normalization. The fully connected layers630may be non-linear and may gradually shrink the high-dimensional output to a dimension of the prediction result (e.g., the classification output640). Thus, the fully connected layers630may also be referred to as a classifier.

The classification output640may indicate a confidence score for each class642based on the input image602. The class642are shown as642a,642b,642c. When the CNN612is trained for abdominal assessment, the classes642may indicate a liver class, an ascending colon class, a descending colon class, an appendix class, a cecum pouch class, and/or a psoas muscle class corresponding to landmarks along a scanning trajectory for the abdominal assessment. A class642indicating a high confidence score indicates that the input image602is likely to include an anatomical object of the class642. Conversely, a class642indicating a low confidence score indicates that the input image602is unlikely to include an anatomical object of the class642.

The CNN612can also output a feature vector650at the output of the last convolutional layer620(N). The feature vector650may indicate objects detected from the input image602. For example, the feature vector650may indicate a liver, a diaphragm, and/or a kidney identified from the image602.

FIG.7is a schematic diagram of a deep learning network training scheme700for providing scanning assistance in an appendicitis assessment, according to aspects of the present disclosure. The scheme700can be implemented by the systems100and500. To train the deep learning network610to provide scanning assistance for an appendicitis assessment, a training data set710(e.g., the image data set140) can be created. The training data set710may include annotated B-mode images. An expert may annotate the B-mode images with labels indicating anatomical objects and/or imaging artifacts. The B-mode images in the training data set710may include annotations for anatomical landmarks that are along a scanning trajectory (e.g., the trajectory450) for appendicitis assessment. For instance, the B-mode images may include annotations for liver, ascending colon, cecum pouch, terminal ileum, appendix/appendicitis, and/or target signatures indicative of appendicitis.

In the illustrated example ofFIG.7, a B-mode image702is captured from a patient704and the image702is annotated by an expert with an annotation730for a mirror artifact, an annotation732for a diaphragm, an annotation734for a liver, and an annotation736for a kidney. The annotated image702is input to the deep learning network610for training. In some instances, the annotations may include bounding boxes in the area where the corresponding anatomical objects reside. The annotations730,732,734, and736serve as the ground truths for training the deep learning network610.

The deep learning network610can be applied to each image702in the data set, for example, using forward propagation, to obtain an output712for the input image702. The training component720adjusts the coefficients of the filters622in the convolutional layers620and weightings in the fully connected layers630, for example, by using backward propagation to minimize a prediction error (e.g., a difference between the ground truth and the prediction result712). The prediction result712may include anatomical objects identified from the input image702. In some instances, the training component720adjusts the coefficients of the filters622in the convolutional layers620and weightings in the fully connected layers630per input image. In some other instances, the training component720applies a batch-training process to adjust the coefficients of the filters622in the convolutional layers620and weightings in the fully connected layers630based on a prediction error obtained from a set of input images.

In some aspects, instead of including bounding boxes and annotations in a training image, the training data set710may store image-class pairs. For instance, each training image may be associated with a specific class (e.g., a liver, an ascending colon, a normal clinical condition, an abnormal clinical condition, a good imaging quality, or a bad imaging quality). The deep learning network610may be fed with the image-class pairs from the training data set710and the training component720can apply similar mechanisms to adjust the weightings in the convolutional layers620and/or the fully-connected layers630to minimize the prediction error between the ground truth (e.g., the specific class in the image-class pair) and the prediction output712.

FIG.8is a schematic diagram of a deep learning network training scheme800for providing scanning assistance in an intussusception assessment, according to aspects of the present disclosure. The scheme800can be implemented by the systems100and500. The scheme800is substantially similar to the scheme700, but may use a different training data set810including annotated images and/or image-class pairs specific to a scanning trajectory (e.g., the trajectory460) for intussusception assessment. For instance, the B-mode images in the training data set810may include annotations or classes for psoas muscle, ascending colon, liver, gall bladder, epigastrium, descending colon, and target signatures of intussusception.

In the illustrated example ofFIG.8, the training data set810includes a B-mode image802is captured from a patient and the image802is annotated by an expert with an annotation830for an iliac crest, an annotation832for a psoas muscle, an annotation834for a transversus abdominal muscle, an annotation836for an internal oblique abdominal muscle, an annotation838for an external oblique abdominal muscle, and an annotation840for a rectus muscle. The training data set810may also include a B-mode image804including a target signature appearance of intussusception as shown by the labels842and844.

The training component820can apply similar mechanisms as in the training component720to adjust the weightings in the convolutional layers620and/or the fully-connected layers630to minimize the prediction error between the ground truth and the prediction output812for each training image (e.g., the images802and804). In some instances, the ground truth may correspond to the annotations of the anatomical objects and the prediction output812may include one or more anatomical objects (e.g., psoas muscle, rectus muscle, liver) identified from an input image. In some other instances, the ground truth may correspond to an image class and the prediction output812may include a predicted class (e.g., a psoas muscle class, a liver class, a normal clinical condition, an abnormal clinical condition, a good imaging quality, or a bad imaging quality).

FIG.9is a schematic diagram of a user interface900for an automated ultrasound scanning assistance system, according to aspects of the present disclosure. In particular, the user interface900can provide scanning assistance to a sonographer for an appendicitis assessment. The scanning may be along a scanning trajectory similar to the scanning trajectory450in an RLQ of a patient. The user interface900may be implemented by the system500and may be displayed on the visual guidance component570ofFIG.5. The user interface900is discussed in relation toFIG.5and may refer to the components shown inFIG.5.

The user interface900incudes a visual guidance map910. The visual guidance map910may illustrate an overall scan pattern (including a colon or large intestine974view of a patient under the scan) jointly with a depiction of the current transducer or probe position and orientation of the probe (shown by the transducer symbol901). The probe position and/or orientation is inferred by the deep learning network582ofFIG.5. At time t1, a sonographer may place a probe510(shown by the transducer symbol901) transversely at an initial location912on a patient's abdomen and capture a first image (e.g., the images512). The deep learning network582may infer from the first input image, whether the first input image is located at a correct initial location or not. The deep learning network582may perform the inference based on one or more anatomical landmarks detected from the first input image. The anatomical landmarks may include to an ascending colon, a transvers colon, a descending colon, a sigmoid colon, and the rectum as shown by the circled numerals 1, 2, 3, 4, and 5, respectively. The one or more detected anatomical landmarks may serve as an input for determining whether the probe is at a correct location or not.

From the initial location912, the deep learning network582infers a steering configuration (e.g., the steering configuration546) to guide the sonographer in sweeping towards the RLQ (e.g., the RLQ420), where landmarks such as a cecum pouch970(e.g., the cecum250) is present. The cecum pouch970serves as a critical landmark to detect the close proximity of the appendix972(e.g., the appendix210). The probe510may continue to capture images as the sonographer sweeps the probe510.

While the sonographer sweeps the probe510, the deep learning network582ensures the received input images include essential landmarks to infer the sweep operation is sufficiently correct. In this regard, the deep learning network582may search for landmarks of the ascending colon (circled numeral 1) from the captures images. If the deep learning network582determines that the landmarks of the ascending colon is not found in the captured images, the deep learning network582may infer a steering configuration for a corrected action and the visual guidance map910may show the action (e.g., in the form of arrows) to guide the user in correcting the position of the probe510. For instance, if the deep learning network582detects that the ascending colon is drifting to the left in the image while the sonographer sweeps the probe510along the ascending colon (i.e., the probe510is drifting to the right of the ascending colon), then the deep learning network582may infer a steering configuration for moving the probe510to the left. The visual guidance map910may show a left arrow to suggest a course correction to the left in accordance with the steering configuration.

When the deep learning network582detected a next landmark (e.g., the cecum pouch970) at time t2, the deep learning network582provides a different course of action. For instance, the deep learning network582may infer a probe steering configuration to physically rotate the probe510by about 90 degrees (as shown by the transducer symbol901b) and move the probe510longitudinally (shown by914). In some other instances, this operation can be achieved by automatic beam steering when the probe510is a 2D array matrix probe instead of having the sonographer to physically maneuver the probe510. The 2D transducer array (e.g., the transducer array112) at the probe510provides capability to select arbitrary imaging planes by appropriately phasing the transducer elements in the array. The 2D probe510can acquire images at various imaging planes and the images can be fed to the deep learning network582to infer a next imaging plane towards the target site where the appendix972is located.

During the longitudinal sweep914, characteristics of the appendix972for appendicitis is inferred from the deep learning network582. During this continuous sweep operation, the acquired images are fed into the auto-capture component550ofFIG.5to conclude whether a sufficient scanning coverage is achieved or not. Alternatively, the deep learning network582can infer whether a sufficient scanning coverage is achieved based on the acquired images. Additionally, the acquired images are fed into the auto-scan setting component560ofFIG.5to automatically adjust the ultrasound signal setting at the probe510to provide on optimal imaging quality for the assessment. Alternatively, the deep learning network582can infer an ultrasound signal setting for adjusting the ultrasound signal at the probe510.

WhileFIG.9illustrates the guidance in the forms of arrows and orientation of the transducer suggesting a required action (e.g., translation and/or rotation) at the current position of the probe510, the user interface900can provide the guidance in any suitable visual form and/or audio form (e.g., voice alerts). For instance, color-coded arrows may be used, where a red-colored arrow may indicate a deviation from the current scanning trajectory and a green arrow may indicate that the probe510is within the scanning trajectory. The deep learning network582may detect the deviation based on a certain landmark identified from an acquired image. The user interface900may further provide a voice alert indicating the deviation and a course correction.

The user interface900can further display one or more captured still image frames920that are indicative of a normal appendix or a suspected abnormal appendix (e.g., with inflammation or fluid-filled). The image frames920may correspond to the images542. The user interface900can further display detected information930, such as the anatomical objects and/or clinical condition identified from the captured images. The display can be in textual form or in the form of annotations overlaid on top of a captured image.

FIG.10is a schematic diagram of a user interface1000for an automated ultrasound scanning assistance system, according to aspects of the present disclosure. In particular, the user interface1000can provide scanning assistance to a sonographer for an intussusception assessment. The user interface1000may be implemented by the system500and may be displayed on the visual guidance component570ofFIG.5. In the illustrated example ofFIG.10, the user interface1000is on the display of a mobile phone. In some other examples, the user interface1000can be displayed on any point-of-care devices. The user interface1000is discussed in relation to theFIG.5and may refer to components shown inFIG.5. The user interface1000is substantially similar to the user interface900. For instance, the user interface1000includes a visual guidance map1010similar to the visual guidance map910with similar landmarks shown by circled numerals 1-5. However, the visual guidance map1010may provide guidance for a different scanning trajectory specific for an intussusception assessment.

For instance, at time t1, the sonographer may place the probe510transversely at an initial location1012in an RLQ of a patient's abdomen (depicted as a transducer symbol1001aon the map1010) and capture a first input image. The deep learning network582ofFIG.5may infer a steering configuration from the first input image to steer the probe towards the RUQ along the ascending colon (circled numeral 1). While the sonographer sweeps the probe510, the probe510continues to acquire images and the images are fed to the deep learning network582.

When the deep learning network582detected a next landmark such as a liver from an acquired image at time t2, the deep learning network582may infer a steering configuration to rotate the probe510by about 90 degrees to a longitudinal direction as shown by the transducer symbol1001b. Alternatively, when the probe510is a 2D probe, this operation can be achieved via beam steering instead of a physical rotation of the probe510as described above with respect toFIG.9. Subsequently, the deep learning network582may infer steering configurations to guide the sonographer to trace the probe510longitudinally till a next landmark (e.g., a spleen and/or a right kidney) is detected.

When the deep learning network582detected the next landmark at time t3, the deep learning network582may infer a steering configuration to rotate the probe510by about 90 degrees as shown by the transducer symbol1001c. The deep learning network582may continue to infer steering configurations based on acquired images to guide the sonographer to trace till the LLQ of the patient covering the entire abdominal topography (from the RLQ to the RUQ to the LLQ).

Similar to the user interface900, the user interface1000can further display one or more captured still image frames that are indicative of a normal clinical condition or the presence of a intussusception. In the illustrated example ofFIG.10, the user interface1000shows an image1020with markings1022at locations with suspected intussusception.

FIG.11is a schematic diagram of a processor circuit1100, according to embodiments of the present disclosure. The processor circuit1100may be implemented in the probe110and/or the host130ofFIG.1. In an example, the processor circuit1100may be in communication with the transducer array112in the probe110. As shown, the processor circuit1100may include a processor1160, a memory1164, and a communication module1168. These elements may be in direct or indirect communication with each other, for example via one or more buses.

The processor1160may include a CPU, a GPU, a DSP, an application-specific integrated circuit (ASIC), a controller, an FPGA, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein, for example, aspects ofFIGS.5-10, and12. The processor1160may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The memory1164may include a cache memory (e.g., a cache memory of the processor1160), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, solid state memory device, hard disk drives, other forms of volatile and non-volatile memory, or a combination of different types of memory. In an embodiment, the memory1164includes a non-transitory computer-readable medium. The memory1164may store instructions1166. The instructions1166may include instructions that, when executed by the processor1160, cause the processor1160to perform the operations described herein, for example, aspects ofFIGS.5-10and12and with reference to the probe110and/or the host130(FIG.1). Instructions1166may also be referred to as code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms “instructions” and “code” may refer to one or more programs, routines, sub-routines, functions, procedures, etc. “Instructions” and “code” may include a single computer-readable statement or many computer-readable statements.

The communication module1168can include any electronic circuitry and/or logic circuitry to facilitate direct or indirect communication of data between the processor circuit1100, the probe110, and/or the display132. In that regard, the communication module1168can be an input/output (I/O) device. In some instances, the communication module1168facilitates direct or indirect communication between various elements of the processor circuit1100and/or the probe110(FIG.1), the probe510(FIG.5) and/or the host130(FIG.1)

FIG.12is a flow diagram of an ultrasound imaging method1200with automatic assistance, according to aspects of the present disclosure. The method1200is implemented by the system100, for example, by a processor circuit such as the processor circuit1100, and/or other suitable component such as the probe110or510, the processor circuit116, the host130, and/or the processor circuit134. In some examples, the system100can include computer-readable medium having program code recorded thereon, the program code comprising code for causing the system100to execute the steps of the method1200. The method1200may employ similar mechanisms as in the systems100and500described with respect toFIGS.1and5, respectively, the configuration600described with respect toFIG.6, the schemes700and800described with respect toFIGS.7and8, respectively, and the user interfaces900and1000described with respect toFIGS.9and10, respectively. As illustrated, the method1200includes a number of enumerated steps, but embodiments of the method1200may include additional steps before, after, and in between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

At step1210, the method1200includes receiving, at a processor circuit (e.g., the processor circuits134and1100) in communication with an ultrasound probe (e.g., the probes110and510) including a transducer array (e.g., the transducer array112), a first image (e.g., the images512) of a patient's anatomy.

At step1220, the method1200includes detecting, at the processor circuit from the first image, a first anatomical landmark at a first location along a scanning trajectory (e.g., the scanning trajectories450and460) of the patient's anatomy.

At step1230, the method1200includes determining, at the processor circuit based on the first anatomical landmark, a steering configuration (e.g., the steering configuration546) for steering the ultrasound probe towards a second anatomical landmark at a second location along the scanning trajectory.

At step1240, the method1200includes outputting, to a display in communication with the processor circuit, an instruction based on the steering configuration to steer the ultrasound probe towards the second anatomical landmark at the second location.

In some instances, the steering configuration includes at least one of a rotation or a translation.

In some instances, the method1200further includes receiving, at the processor circuit from the ultrasound probe, a second image of the patient's anatomy at the second location, the second image including the second anatomical landmark. The method1200further includes determining, at the processor circuit based on the second image, a beam steering angle to steer ultrasound beams of the transducer array towards a third anatomical landmark at a third location along the scanning trajectory. The method1200further includes outputting, to a controller in communication with the processor circuit and the ultrasound probe, an instruction to configure the transducer array based on the beam steering angle. The controller may be similar to the probe controller150and/or the auto-beam steering component590.

In some instances, the method1200further includes determining, at the processor circuit, an ultrasound signal adjustment for the transducer array based on the first image. The method1200further includes outputting, to a controller in communication with the processor circuit and the transducer probe, an instruction to configure the transducer array based on the ultrasound signal adjustment. In some instances, the ultrasound signal adjustment is associated with at least one of a signal gain or an imaging depth of field. The controller may be similar to the probe controller150and/or the auto-scan setting component560.

In some instances, the method1200further includes receiving, at the processor circuit from the transducer array, a second image of the patient's anatomy at the second location along the scanning trajectory. The method1200further includes determining, at the processor circuit, that the second image includes an anatomical feature representative of a clinical condition. The method1200further includes storing, at a memory (e.g., the memory138) in communication with the processor circuit, the second image based on determining that the second image includes the anatomical feature representative of the clinical condition.

In some instances, the step1240includes outputting, to the display, a map of the scanning trajectory and at least one of a visual motion indicator with respect to the scanning trajectory based on the instruction, a location of the transducer array with respect to the scanning trajectory, or an orientation of the transducer array with respect to the scanning trajectory, for example, as shown in the visual guidance maps910or1010.

In some instances, the patient's anatomy includes an abdominal region of the patient, and wherein the scanning trajectory traverses at least one of a RUQ (e.g., the RUQ410), a RLQ (e.g., the RLQ420), a LUQ (e.g., the LUQ430), or a LLQ (e.g., the LLQ440) of the patient's abdominal region.

In some instances, the scanning trajectory is associated with an appendicitis examination, and wherein the first anatomical landmark includes at least one of a liver, an ascending colon, a cecum pouch, a terminal ileum, an appendix, or an anatomical characteristic of an appendicitis.

In some instances, the scanning trajectory is associated with an intussusception examination, and wherein the first anatomical landmark includes at least one of a psoas muscle, an ascending colon, a liver, a gallbladder, an epigastrium, a descending colon, an anatomical characteristic of an intussusception.

Aspects of the present disclosure can provide several benefits. For example, the use of a deep learning-based framework for automated anatomical landmark detection and probe steering motion prediction can provide a clinician with systematic scanning guidance, reducing ultrasound examination time and/or user-dependency. Thus, the disclosed embodiments may increase the accuracy of clinical assessments (e.g., appendicitis and intussusception assessments) and improve work flow efficiency.

Persons skilled in the art will recognize that the apparatus, systems, and methods described above can be modified in various ways. Accordingly, persons of ordinary skill in the art will appreciate that the embodiments encompassed by the present disclosure are not limited to the particular exemplary embodiments described above. In that regard, although illustrative embodiments have been shown and described, a wide range of modification, change, and substitution is contemplated in the foregoing disclosure. It is understood that such variations may be made to the foregoing without departing from the scope of the present disclosure. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the present disclosure.