Patent Description:
Three dimensional ultrasound imaging has been widely used in the field of medicine. There are usually three ways: electronic scanning, mechanical scanning and manual scanning. Mechanical scanning is to use the motor to drive the ultrasonic probe for scanning, with the advantage of good repeatability, but only suitable for small-scale scanning, such as fetus observation. Electronic scanning can give real-time three-dimensional images, such as the heart, but it is only suitable for scanning in a smaller range, and it needs to use two-dimensional transducer display, making the cost quite expensive. Manual scanning refers to holding the ultrasonic probe by an operator to scan the interested area of human body or animal, recording the three-dimensional spatial position and angle of each ultrasonic image by a spatial positioning system, and then carrying out three-dimensional image reconstruction. Its advantage is that it can do a large range of scanning, but it requires manual scanning.

In recent years, the miniaturization of ultrasound imaging system has developed rapidly. At present, there are many different types of handheld ultrasound systems. By using the handheld ultrasound system, the portability can be greatly improved so that ultrasound imaging can be applied in more fields. However, at present, there is still no handheld three-dimensional ultrasound imaging system in the market, because there are some difficulties in its implementation, especially the palmtop 3D ultrasound imaging system for manual scanning of a large range of human body. Because the traditional ultrasound imaging system requires a large spatial positioning system, which is not suitable for the practical palmtop application. For example, the most commonly used electromagnetic positioning device needs an external transmitter and a spatial positioning sensor placed on the ultrasound probe. It is very difficult to make such a system completely into the palmtop portable system.

Therefore, how to improve the huge spatial positioning system and make the portable handheld three-dimensional ultrasound imaging system widely used has become an urgent technical problem in the industry.

<CIT> discloses a user interface for ultrasound scanning system comprising another sensor that may be included in the device is a camera. The camera may be positioned to record a digital image of the skin surface during an ultrasound scan when the handheld ultrasound probe is placed for use against the skin surface of the target. The camera also may be positioned to obtain a pose of the handheld ultrasound probe as the ultrasound transducer scans the target. The camera may be mechanically coupled to the ultrasound transducer. In one aspect, the camera may be rigidly mounted to the ultrasound transducer and directed toward the skin surface (when positioned for use) in order to capture images of the skin surface and/or a target adhered to the skin surface. It is further disclosed that the system generally includes the handheld ultrasound probe to capture one or more ultrasound images of a target through a skin surface in a scan, a fiducial marker applied to the skin surface of the target and a camera. Further, the ultrasound probe may also include a sensor system configured to obtain a pose of the ultrasound probe. The sensor system may include the camera and a lighting source to capture image of the fiducial marker and or other visible features to obtain camera motion data. Further it is disclosed that a fiducial marker with predetermined features such as predetermined dimensions may be applied to the skin surface of a target. The fiducial marker may be placed in any suitable location. In one embodiment, the fiducial marker is preferably positioned at or near the location where the ultrasound probe will contact the surface so that the camera has a clear view of the fiducial marker. The camera may capture a digital image of the target, or the skin surface of the target. The digital image may include features of the skin surface and/or the fiducial marker where the fiducial marker is within a field of view of the camera. Thus, the fiducial marker is used with the camera to obtain camera motion data to position the probe.

<CIT> discloses a portable ultrasonic imaging system.

<CIT> discloses systems and methods for remote graphical feedback of an ultrasound scanning technique of a local operator.

<CIT> discloses intraoral scanning using ultrasound and optical scan data.

<CIT> discloses systems and methods for ultrasound imaging using an inertial reference unit.

<CIT> discloses a device and method for generating a 3D image based on 2D ultrasound data.

Starting from <CIT>, the object of this invention may be seen in providing a handheld three-dimensional ultrasound imaging system and a corresponding method which do not require a camera for positioning.

This object is achieved by the subject matter of the independent claims, which relates to embodiment <NUM> disclosed below. All other embodiments are exemplary and do not form part of the invention.

Preferred embodiments of the invention are mirrored by the dependent claims.

By means of the handheld three-dimensional ultrasound imaging system and method of the present application, the large spatial positioning system in an existing three-dimensional ultrasound imaging system is changed into a portable spatial positioning system that can be used at any time, so that handheld three-dimensional ultrasound imaging can be widely applied.

As shown in <FIG>, the present application discloses a handheld three-dimensional ultrasound imaging system, comprising a handheld ultrasound probe <NUM>; a display, control and processing terminal <NUM>, connected to the handheld ultrasound probe <NUM> wiredly or wirelessly; a handheld three-dimensional spatial positioning system <NUM>, connected to the handheld ultrasound probe <NUM>, moving with the movement of the handheld ultrasound probe <NUM>, connected to the display, control and processing terminal <NUM> wiredly or wirelessly. In <FIG>, the handheld three-dimensional spatial positioning system <NUM> is mounted on the handheld ultrasound probe <NUM>. In other embodiments, as long as it is connected with the handheld ultrasound probe <NUM>, it can move with the movement of the handheld ultrasound probe <NUM>, without necessarily being installed on the handheld ultrasound probe <NUM>, and the specific setting mode is not limited here. The display, control and processing terminal <NUM> of the application can be a palmtop terminal or a desktop terminal, such as a laptop, etc., which can be connected with the handheld ultrasound probe <NUM> by wireless or wired means. The display, control and processing terminal <NUM> stores three-dimensional imaging, image processing and three-dimensional display algorithms, which directly analyzes and processes the image and data information returned by the handheld ultrasound probe <NUM>, and displays three-dimensional images.

As shown in <FIG>, since the handheld three-dimensional ultrasound imaging system is required to be small in size and portable, in order to reduce the dimension of the handheld three-dimensional ultrasound imaging system, the handheld three-dimensional ultrasound imaging system of the present application further includes a cloud database <NUM> and a similar processing system. Thus, the display, control and processing terminal <NUM> transmits the three-dimensional position information, angle and reconstruction result to the cloud database <NUM> for storage. In the cloud database <NUM>, the reconstruction results can be classified and stored, for example, the time can be classified and stored by the name of the customer, so as to facilitate the user to compare the changes of the part to be tested <NUM> in different time periods, or the classification and storage can be carried out by the name of different diseases, so that the user can refer to the changes of the part to be tested <NUM> to be detected by other users.

Further, in order to make the handheld three-dimensional ultrasound imaging system more miniaturized and improve its portability, only simple 3D imaging, image processing and 3D display algorithms are stored in the display, control and processing terminal <NUM>, and the image and data information are simply analyzed and processed. The display, control and processing terminal <NUM> is connected to the cloud database <NUM> and similar processing systems through the network, Bluetooth and other ways. In the cloud database <NUM>, more advanced and more complex 3D imaging, image processing and 3D display algorithms can be stored. The display, control and processing terminal <NUM> uploads the simple processed information to the cloud database <NUM> for analysis and processing. The cloud database <NUM> transmits the results after analysis and processing back to the display, control and processing terminal <NUM> of the handheld ultrasonic instrument for display or further processing. Even, the display, control and processing terminal <NUM> may not store three-dimensional imaging, image processing and three-dimensional display algorithm, just upload the image and data information returned by the handheld ultrasound probe <NUM> directly to the cloud database <NUM>, and process, analyze and process through the three-dimensional imaging, image processing and three-dimensional display algorithm in the cloud database <NUM>. After that, the information is transmitted back to the display, control and processing terminal <NUM> for display. Similarly, in the cloud database <NUM>, the reconstruction results can still be classified and stored, so that customers can retrieve the data of the reconstruction results from the cloud database <NUM> for query. The cloud database <NUM> can be a remote storage and computing device.

The handheld three-dimensional spatial positioning system <NUM> in the application is a device convenient for moving and installing, which is connected with the handheld ultrasound probe <NUM>, and can move with the movement of the handheld ultrasound probe <NUM>. The handheld three-dimensional ultrasound imaging system can directly obtain the 3D spatial position of the handheld ultrasound probe <NUM> through the handheld three-dimensional spatial positioning system <NUM>, without any other positioning system. Preferably, the handheld three-dimensional spatial positioning system <NUM> is arranged inside the handheld ultrasound probe <NUM>, so that when the handheld three-dimensional ultrasound imaging system of the present application is applied, there will not be a non portable positioning system to affect the portability of the handheld three-dimensional ultrasound imaging system. There are six embodiments of how the handheld three-dimensional spatial positioning system <NUM> obtains the 3D position information and angle information of the handheld ultrasound probe <NUM>.

As shown in <FIG>, in order to improve the accuracy of positioning, the handheld three-dimensional ultrasound imaging system can further include a positioning reference device <NUM>, which is located outside the handheld ultrasound probe <NUM> and is used to provide positioning reference for the handheld three-dimensional spatial positioning system <NUM>.

The handheld three-dimensional spatial positioning system <NUM> includes micro inertial sensors such as accelerometers and angular velocity meters installed on the handheld ultrasound probe <NUM>, which are used to obtain the acceleration and angular acceleration values of the handheld ultrasound probe <NUM>, so as to calculate the moving distance and rotation angle of the handheld ultrasound probe <NUM>, and then independently obtain the 3D spatial position of the handheld ultrasound probe <NUM>.

The handheld three-dimensional spatial positioning system <NUM> includes one or more cameras installed on the handheld ultrasound probe <NUM> and micro inertial sensors such as accelerometers, angular velocity meters installed in the handheld ultrasound probe <NUM>. The positioning reference device <NUM> is an external environment. The camera is used to obtain the image of the surrounding environment, such as the grid on the ceiling, etc. according to the obtained image changes, the position and angle of the handheld ultrasound probe <NUM> can be calculated, and special graphics can also be simply added in the environment to facilitate detection. Accelerometers and angular velocity meters are used to obtain the acceleration and angular acceleration values of the handheld ultrasound probe <NUM>, so as to calculate the moving distance and angle of the handheld ultrasound probe <NUM>. The combination of accelerometer, angular velocity meter and camera makes the positioning more accurate. Before using the handheld three-dimensional spatial positioning system <NUM>, it is necessary to let the handheld ultrasound probe <NUM> move a known distance or rotate a known angle to determine the parameters needed in the positioning algorithm.

As shown in <FIG>, the handheld three-dimensional spatial positioning system <NUM> is a micro inertial sensor, such as accelerometer and gyroscope, which are installed on the handheld ultrasound probe <NUM>. The handheld three-dimensional ultrasound system further includes a positioning reference device <NUM>, which is a small reference system installed on or near the scanning object, i.e. installed outside the handheld ultrasound probe <NUM>, and is used to provide positioning reference for the handheld three-dimensional spatial positioning system <NUM>. The miniaturized reference system is a miniaturized electromagnetic transmitter installed on the scanning object. In addition to the electromagnetic transmitter, the sound or light transmitting and receiving system can also be used, that is, the miniaturization of the traditional light, sound and electromagnetic handheld three-dimensional spatial positioning system. Accelerometers and gyroscope are used to obtain the acceleration and angular acceleration values of the handheld ultrasound probe <NUM>, so as to calculate the moving distance and angle of the handheld ultrasound probe <NUM>. The combination of accelerometer, angular velocity meter and small reference system makes the positioning more accurate.

In addition, the small reference system can also be one or more micro cameras placed on the scanning body, recording the rotation angle of the micro camera to track the movement and angle of the probe.

The handheld three-dimensional spatial positioning system <NUM> in the fourth embodiment is a micro inertial sensor such as an accelerometer, a gyroscope, etc. The handheld three-dimensional ultrasound system further includes a positioning reference device <NUM>, which is the ultrasound image itself.

As shown in <FIG>, when the handheld ultrasound probe <NUM> moves, the left and right ultrasound images in <FIG> are obtained successively. The two ultrasound images have continuity and some overlapped parts, that is, the content of the ultrasound images obtained sequentially has great similarity. When the image sampling speed is very high, but the moving speed is not very fast, the difference distance d between the two images can be obtained by the way of image matching, and the moving distance can be obtained by the difference between the successively obtained ultrasound images. In the same way, the rotation angle of the handheld ultrasound probe <NUM> in this plane can also be obtained. However, when the ultrasound image itself is used alone, the limitation is that the handheld ultrasound probe <NUM> can only move or rotate in one direction. When the handheld ultrasound probe <NUM> moves or rotates in the reverse direction, it is easy to cause the calculation error of the collected three-dimensional position information or angle information.

Therefore, in this embodiment, micro inertial sensors such as accelerometers and angular velocity meter are used in combination with ultrasound images, and micro inertial sensors such as accelerometers and angular velocity meter are used to obtain the acceleration and angular acceleration values of the handheld ultrasound probe <NUM>, so as to calculate the moving distance and angle of handheld ultrasound probe <NUM>, which can supplement the data obtained from the images, making the positioning method more accurate.

The difference between the fifth embodiment and the fourth embodiment is that the handheld ultrasound probe of the fifth embodiment uses a compound probe, that is, a sub probe in different directions is installed in one handheld ultrasound probe <NUM>, which is used to measure the moving distance and rotation angle of the handheld ultrasound probe <NUM> in both directions at the same time. The handheld three-dimensional spatial positioning system <NUM> is still a micro inertial sensor such as accelerometer and gyroscope. It is used to obtain the acceleration and angular acceleration values of the handheld ultrasound probe <NUM>, and calculate the moving distance and angle of the handheld ultrasound probe <NUM>. Thus, the handheld three-dimensional spatial positioning system <NUM> can detect the amount of movement and rotation in the third direction besides the direction provided by the compound probe, and then calculate the 3D position information and angle of the handheld ultrasound probe <NUM> more accurately.

As shown in <FIG>, the handheld three-dimensional ultrasound imaging system further includes a positioning reference device <NUM>, in this embodiment, the positioning reference device <NUM> is a localization image <NUM> set on the surface of the scanning object. The handheld three-dimensional spatial positioning system <NUM> includes a camera <NUM> mounted on the handheld ultrasound probe <NUM>. When the handheld ultrasound probe <NUM> moves, the camera <NUM> tracks the moving distance and rotation angle of the handheld ultrasound probe <NUM> according to the change of the localization image <NUM>. Preferably, the handheld three-dimensional spatial positioning system <NUM> also includes micro inertial sensors such as accelerometers and gyroscope installed on the handheld ultrasound probe <NUM> for further providing information about the moving distance and rotation angle of the handheld ultrasound probe <NUM>. The localization image <NUM> may be a specially designed image temporarily pasted on the surface of the scanning object, and they are pasted beside the part to be tested <NUM> of the object to be scanned, so as to avoid the interference of the localization image <NUM> on the ultrasonic signal. In this embodiment, the localization image <NUM> is a lattice attached to the part to be tested <NUM>, and the distance between the points in the lattice is a pre-designed value, which is known. Preferably, the lattice can also be designed as points with large and small intervals, which can be used to provide a clearer positioning reference.

In this embodiment, the camera <NUM> is used to record the localization image <NUM> for positioning. In other embodiments of the application, other similar methods can be applied, for example, the localization image <NUM> with the characteristics of sound, light, electrical, magnetic, etc. is designed and pasted on the scanning object, and a detector is installed on the handheld ultrasound probe <NUM> to record the localization image <NUM> located outside the part to be tested <NUM> for positioning.

The difference between the seventh embodiment and the sixth embodiment is that the localization image <NUM> of the sixth embodiment is temporarily attached to the part to be tested <NUM> of the object to be scanned, that is, the ultrasound will not scan the localization image <NUM> to avoid interference of the localization image <NUM> with the ultrasonic signal. However, in the sixth embodiment, the interference of the localization image <NUM> with the ultrasonic signal is used for positioning.

As shown in <FIG>, the localization image <NUM> is a lattice, of course, in other embodiments of the application, the localization image <NUM> can also be of other shapes, such as lattice, wave shape, etc., as long as it can provide positioning reference. The localization image <NUM> in the embodiment is pasted on the part to be tested <NUM>, and the distance between the points in the lattice is a pre-designed value, which is known. Preferably, the setting mode can be a mode with one big point for every five small points to provide positioning information more clearly. Further, according to the material of the localization image <NUM>, each point on the lattice has different degrees of influence on the ultrasonic signal. As shown in <FIG>, the reflection signal can be a point, as shown in <FIG>, and the reflection signal can also be a point with a shadow area, which is used to distinguish the position of each row or column of points, so as to make the positioning more accurate. Preferably, the localization image <NUM> is integrated with the ultrasonic coupling paste, which makes the use and attachment on body surface easier and the operation process more concise.

Therefore, in this embodiment, there are two positioning methods. The first method is to extract the information of the localization image <NUM> from the obtained ultrasound image as positioning information, then recover the ultrasound image to a state without interference of the localization image <NUM> after image processing, and then reconstruct the three-dimensional image of the ultrasound image.

Another positioning method,which has not been claimed, is to scan the part to be tested <NUM> once when the localization image <NUM> is attached on the part to be tested <NUM> to obtain the first ultrasound image, the scope of the first ultrasound image may include the localization image <NUM> and the scope beyond the localization image <NUM>; then take away the localization image <NUM>, and then scan again to obtain the second ultrasound image; the display, control and process terminal determines the position of the localization image <NUM> relative to the second ultrasound image according to the obtained first ultrasound image as a reference, so as to obtain a three-dimensional image which is completely free from the influence of the localization image <NUM> and has positioning information of the localization image <NUM>.

As shown in <FIG>, the above localization image <NUM> is used in the embodiment to interfere with ultrasonic wave for positioning. In other embodiments of the application, other similar methods can be applied, such as designing localization image <NUM> with optical, electrical, magnetic and other characteristics, attaching it to the scanning object, and installing optical, electrical, magnetic and other detectors on the handheld ultrasound probe <NUM> as the palm of the system The localization image <NUM> is detected by the handheld three-dimensional spatial positioning system <NUM>, and the installation mode of the handheld three-dimensional spatial positioning system <NUM> is shown in the figure.

It can be understood that in the first to seventh embodiments, all the handheld three-dimensional spatial positioning system <NUM> installed on the handheld ultrasound probe <NUM>, i.e. camera, sound, light, electrical, magnetic detector, etc., may not be installed on the handheld ultrasound probe <NUM>, as long as they can move with the movement of the handheld ultrasound probe <NUM>, there is no limit here.

As shown in <FIG>, the present application further discloses a handheld three-dimensional ultrasound imaging method, comprising the following steps:.

In the handheld three-dimensional ultrasound imaging method, the step S2 comprises the following steps:
S2. <NUM> arranging a positioning reference device <NUM> on the scanning objects for providing positioning reference for the handheld three-dimensional spatial positioning system <NUM>.

The positioning reference device <NUM> is an electrical, magnetic, acoustic, optical and other transmitters installed on the scanning object, and the transmitted electrical, magnetic, acoustic, optical and other signals can be received by the corresponding receiver installed on the handheld ultrasound probe <NUM> for positioning reference; or place the localization image <NUM> with electrical, magnetic, acoustic and optical characteristics on the surface of the scanning object, so that it can be detected by the ultrasonic transducer on the handheld ultrasound probe <NUM>, or the electrical, magnetic, acoustic and optical detectors, and then used for positioning. For specific embodiments, please refer to the first to sixth embodiments, which will not be described here.

In the handheld three-dimensional ultrasound imaging method, the step S3 comprises the following steps:
S3. <NUM> performing 3D image reconstruction to the ultrasound image and the three-dimensional position and angle information and displaying through the display, control and processing terminal <NUM>.

In order to realize the miniaturization of the handheld three-dimensional ultrasound imaging system, when the handheld three-dimensional ultrasound imaging system further includes the cloud database <NUM> for storing data, step S3 further includes:
S3. <NUM> The display, control and processing terminal <NUM> transmits the reconstruction result to the cloud database <NUM> for storage.

In the cloud database <NUM>, the reconstruction results can be classified and stored, for example, the time can be classified and stored by the name of the customer, so as to facilitate the user to compare the changes of the part to be tested <NUM> in different time periods, or the classification and storage can be carried out by the name of different diseases, so that the user can refer to the changes of the part to be tested <NUM> by other users.

In order to realize further miniaturization and portability of the handheld three-dimensional ultrasound imaging system, the handheld three-dimensional ultrasound imaging system further includes a cloud database <NUM> for data processing, and the steps S3 of the imaging method include:.

Similarly, in the cloud database <NUM>, the reconstruction results can be classified and stored, so that customers can retrieve the data of the reconstruction results from the cloud database <NUM> for query.

The handheld three-dimensional spatial positioning system <NUM> in the method can be any of the handheld three-dimensional spatial positioning system in the first to seventh embodiments, and the positioning reference device <NUM> of the application can be any of the positioning reference devices in second to seventh embodiments; when the positioning reference device <NUM> is the material shown in the seventh embodiment, which is a material that can interfere with ultrasound imaging, and is disposed on the part to be tested <NUM>, the step S3 of the imaging method further includes the following steps:.

In the seventh embodiment, there is another positioning method. Step S1 of the imaging method further includes the following steps:.

Claim 1:
A handheld three-dimensional ultrasound imaging system, comprising
a handheld ultrasound probe (<NUM>), used for scanning and obtaining an ultrasound image;
a display, control and processing terminal (<NUM>), connected to the handheld ultrasound probe (<NUM>) wiredly or wirelessly and configured to display the ultrasound image;
a handheld three-dimensional spatial positioning system (<NUM>), connected to the handheld ultrasound probe (<NUM>), moving with movement of the handheld ultrasound probe (<NUM>), connected to the display, control and processing terminal (<NUM>) wiredly or wirelessly, and used for obtaining a three-dimensional position and angle information of the handheld ultrasound probe (<NUM>);
a location image (<NUM>) , located outside the handheld ultrasound probe (<NUM>), used for providing positioning reference for the handheld three-dimensional spatial positioning system (<NUM>);
characterized in that
the localization image (<NUM>) is arranged on a part to be tested (<NUM>) of an object to be scanned and comprises a lattice, wherein the distance between points in the lattice is a predesigned value, which is known, and wherein the reflection signal of each point is a point or a point with a shadow;
the display, control and processing terminal (<NUM>) is configured to extract information of the localization image (<NUM>) from the obtained ultrasound image as positioning information; and
the display, control and processing terminal (<NUM>) is configured to recover the ultrasound image to a state without interference from the localization image (<NUM>) after image processing, and further configured to perform 3D image reconstruction of the ultrasound image.