Patent Description:
Pelvic floor dysfunctions (PFDs) is a common medical condition that affects one third of women in China and one quarter of women in the United States. The disease is related to abnormal spasms or control issues in the muscle that support the pelvic organs. Common symptoms of pelvic floor dysfunctions include pelvic organ prolapse, urinary incontinence and faecal incontinence.

In spite of the prevalence of pelvic floor dysfunction, the condition is often not reported by patients until severe symptoms have shown. Recently, post-partum screening has become more common to diagnose the condition in the earlier stages. Physical examination is widely used as an initial approach for assessing PFDs, and can be used to diagnose approximately <NUM>% of conditions. However, results from physical exams are highly subjective and descriptive, experience-dependent, and have limited use for determining the severity of the disease. In order to address this, imaging methods such as ultrasound are becoming more widely used to assess a subject.

Ultrasound and magnetic resonance (MR) are two commonly used imaging modalities for assessing pelvic floor structures and functions. Ultrasound typically serves as the first-line imaging modality. Compared to MR, ultrasound has the advantages of real-time imaging, and is widely available, low cost, and easy-to-use, which are beneficial in PFD diagnosis. Observations and measurements of the movement of pelvic organs during muscle contraction and Valsalva maneuver are important for assessing pelvic organ prolapse and have direct indication for urinary incontinence, fecal incontinence and other symptoms.

Pelvic floor ultrasound (PFUS) has proven useful for evaluating the pelvic floor including the urethra, bladder, vagina, cervix, anorectum, levator ani and sphincter. Both descriptive observations and quantitative measurements are taken during a PFUS examination for a comprehensive assessment of the pelvic floor of a subject.

<CIT> disclosed a pelvic floor measurment method where pelvic floor key part points can be automatically recognized on the utrasound pelvic floor image. <CIT> disclosed an input unit that inputs feature points of a target part and a contour extraction unit that extracts a contour along the shape of the target part using the feature points. <CIT> disclosed a handheld ultrasound imaging apparatus, including a display and a pluarity of user inerface objects, wherein the pointer of the user interfance can be set as a caliper for distance measurement.

The current methods available for assessing pelvic floor ultrasound images involve a clinician identifying relevant points corresponding to well-defined landmarks, or anatomical features or sub-features, in an ultrasound image which may then be used to perform a measurement. The clinician may click on the identified points and indicate the points that have been identified to an interface, along with a desired measurement, so that a computing device (or, for example, software) can then make the calculation or perform the measurement using the points. However, a user may be required to identify the same point corresponding to the same anatomical feature to make a new measurement, even if the point has been previously used to perform a different measurement, and the desired measurement must also be indicated along with the identified points. Furthermore, current methods may require a user to indicate points in a particular order.

In other existing systems, the clinician may be required to indicate points in a specific order shown on the user interface to take a measurement corresponding to a particular workflow. Such interfaces may not allow the clinician to skip or change the order of the inputs required for the measurements. However, in a routine workflow, the clinician may not always need to measure all the parameters set by an interface. Furthermore, these systems may only allow a limited number of measurements to be performed with a limited number of identified points.

It is therefore desirable to improve the efficiency of the processes by which images relating to a pelvic floor of a subject can be analysed, and to improve the flexibility of such processes.

According to an aspect, there is provided an apparatus for use in processing image data relating to a subject, the apparatus comprising a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, cause the processor to, responsive to receiving input indicating a point in an image corresponding to an anatomical feature or sub-feature of the subject, determine if sufficient points to perform a measurement have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement; and if sufficient points to perform a measurement have been received, perform the measurement.

Advantageously, this enables a more efficient process whereby measurements are performed automatically once all features and/or sub-features required to perform a measurement have been identified. The proposed solution may perform measurements automatically once all the required points have been collected, and may use the same point for different measurements where possible. This reduces the time taken for measurements to be obtained.

The pre-stored anatomical data may therefore be a data set which comprises a list of anatomical features or sub-feature, where groups of features from the list may be usable, or required, to perform particular measurements. It will be appreciated that the same anatomical feature or sub-feature may be used to perform different measurements, e.g. the same anatomical feature or sub-feature may be used to perform more than one measurement. The determining may involve comparing any received points (e.g. any points that have been indicated by the user for the same image data) to the pre-stored anatomical data to determine if the points which have been indicated can be used to perform a measurement, for example each time a new point is indicated.

The processor may be further caused to determine to which anatomical feature or sub feature the point corresponds based on received input from a user indicating the anatomical feature or sub feature to which the point corresponds, or by classifying the point into a class corresponding to the anatomical feature or sub feature.

The processor may be further caused to determine to which anatomical feature or sub feature the point corresponds by classifying the point into a class corresponding to an anatomical feature using a first portion of the image around the point and using the position of the point relative to the image, and if the class into which the point is classified comprises sub-classes corresponding to sub-features of the anatomical feature, classify the point into a sub-class of the class using a second portion of the image around the point.

The processor may be for use in processing image data relating to the pelvic floor of a subject. The image data may be image data relating to the pelvic floor of the subject. The anatomical feature or sub-feature can be a feature or sub-feature usable for assessing the pelvic floor of the subject. The measurement can be a measurement usable for assessing the pelvic floor of the subject. It will be appreciated that examples herein are described in relation to processing image date relating to the pelvic floor or to measurement usable for assessing the pelvic floor, however, the proposed technical solution could be equally applied to other measurements or clinical applications, especially where more than one anatomical feature or sub-features (e.g. landmarks) are to be indicated or identified for measurements. For example, it can be applied to fetal scanning, where measurments can include any one or more of biparietal diameter, head circumference, abdominal circumference, femoral diaphsisi length as well as gestational age, fetal weight estimated based on one or more of the aforementioned measurements, and the corresponding anatomical features or sub-features can include any one or more of outer edge of the proximal skull, inner edge of the distal skull, ends of the femoral shaft along the long axis, the junction of the umbilical vein, poral sinums and fetal stomach, etc..

The image data may be any one of an ultrasound image and megnetic resonance image. The image data may be image data relating to a plane of a subject. The plane may be a sagittal plane, a coronal plane, an axial plane, (or any off-axis plane) for example. The sagittal plane may be a mid-sagittal plane. It will be appreciated that examples herein are described in relation to the mid-sagittal plane, however, the proposed technical solution could be applied to any planes which are used in the analysis of a pelvic floor of a subject. Furthermore, while the examples herein are described in relation to ultrasound imaging systems, it will be appreciated that the proposed technical solution herein may be equally applicable for magnetic resonance imaging (MRI) and therefore MRI systems may also embody the proposed technical solution herein. The methods herein may be equally applicable for images acquired by other existing or future developed imaging technology as well, such as computer tomographic (CT) imaging.

Advantageously, the proposed solution may automatically classify points into classes relating to a class or a sub-class, thereby reducing the requirement for user input and increasing the speed at which a points required for measurement are collected. Therefore, the time taken to process image data for diagnosis may be reduced.

The proposed solution may further comprise performing a measurement relating to the anatomical feature or sub feature corresponding to the class or sub-class into which the point is classified, and wherein the measurement is performed using the position of the point in the image. The method may further comprise classifying a plurality of points into classes and sub-classes, and performing the measurement once points have been identified as belonging to classes and/or sub-classes corresponding to all the anatomical features and/or sub-features required for the measurement to be performed.

According to a further aspect, there is provided an ultrasound imaging system comprising the apparatus as described herein and an ultrasound transducer with which to obtain image data relating to the image.

According to a further aspect there is provided a computer implemented method for use in processing image data relating to a subject, the method comprising: responsive to receiving input indicating a point in the an image corresponding to an anatomical feature or sub-feature, determining if sufficient points to perform a measurement have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement; and if sufficient points to perform a measurement have been received, performing the measurement.

According to a further aspect there is provided a computer program product comprising computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the methods described herein.

Example embodiments will now be described, by way of example only, with reference to the following drawings, in which:.

In the standard practice of pelvic floor ultrasound, there are five steps in image acquisition and analysis including (the order of the steps may change in clinical practice):.

The current methods available for assessing pelvic floor ultrasound images may involve a clinician identifying relevant points corresponding to landmarks in an ultrasound image which can be used in order to perform a measurement. The clinician may click on the identified points and indicate the points that have been identified to an interface, along with a desired measurement, so that a computing device can then make the calculation or perform the measurement using the points.

Table <NUM> below illustrates the imaging steps and corresponding measurements to be performed for the each of the five imaging steps outlined above. Table <NUM> outlines the imaging step, the measurement along with the measurement type, and the points required to take a measurement. Table <NUM> also shows calculations that can be performed using the measurements. For example, for the first two imaging steps (of the mid-sagittal plane at rest and when the subject is performing the Valsalva maneuver), <NUM> measurements are performed in the first step, and <NUM> measurements are performed in the second step as shown in the table below. As can be seen from this table, in the same image, the same points may be used for different measurement (as an example, it can be seen that the symphysis pubis (SP) (e.g. a horizontal line through the SP) is used for measurement of the bladder to the SP, the uterus to the SP, the rectal ampulla to the SP, and so on, in the first imaging step). In current systems, a user may be required to identify a point corresponding to the SP (e.g. indicate a point on a horizontal line through the SP) each time a new measurement is to be performed.

Imaging steps and corresponding measurements for analysis of the pelvic floor.

In order to reduce the length of time required to obtain the required measurements and provide a more efficient system, methods described herein are provided in which when a user indicates a point in an image, a measurement can be performed once sufficient points to perform the measurement have been obtained. Furthermore, a point relating to an anatomical features or sub feature can be re-used for different measurements, so it is not necessary for a clinician to re-click on a point. The points may be stored so that it is not necessary to repeatedly indicate the same anatomical feature or sub feature in an image.

Furthermore, methods are described herein where the anatomical feature or sub feature corresponding to the point can be automatically classified, and when all points corresponding to a measurement have been classified, the measurement may be performed using the points.

As described above, the apparatus and methods described herein allow for improved efficiency of processing of image data relating to a pelvic floor of a subject.

Turning now to <FIG>, which shows an apparatus <NUM> for use in processing image data relating to a pelvic floor of a subject, according to some embodiments herein. Generally, the apparatus may form part of a computer apparatus or system e.g. such as a laptop, desktop computer or other computing device. In some embodiments, the apparatus <NUM> may form part of a distributed computing arrangement or the cloud.

Generally, the apparatus may be adapted or configured to perform any of the embodiments of the methods described herein, such as the method <NUM> as described below with respect to <FIG>.

The apparatus comprises a memory <NUM> comprising instruction data representing a set of instructions <NUM> and a processor <NUM> (e.g. processing circuitry or logic) configured to communicate with the memory and to execute the set of instructions <NUM>. Briefly, the set of instructions <NUM>, when executed by the processor <NUM>, cause the processor <NUM> to, responsive to receiving input indicating a point in the an image corresponding to an anatomical feature or sub-feature, determine if sufficient points to perform a measurement have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement; and if sufficient points to perform a measurement have been received, perform the measurement.

As described above, current methods for assessing pelvic floor ultrasound images may require a user to indicate the same points in an image when the same point is required for a different measurement. Furthermore, current methods may require selection of a measurement name to enable a calculation across measurements, and there may be little flexibility in the order in which points are required to be indicated by the user or the measurements that are performed. The proposed apparatus herein advantageously improves the efficiency of processing of image data.

In more detail, the processor <NUM> may comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the apparatus <NUM> in the manner described herein. In particular implementations, the processor <NUM> can comprise a plurality of software and/or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein. The processor <NUM> can comprise one or more processors, processing units, multi-core processors and/or modules that are configured or programmed to control the apparatus <NUM> in the manner described herein. In some implementations, for example, the processor <NUM> may comprise a plurality of (for example, interoperated) processors, processing units, multi-core processors and/or modules configured for distributed processing. It will be appreciated by a person skilled in the art that such processors, processing units, multi-core processors and/or modules may be located in different locations and may perform different steps and/or different parts of a single step of the method described herein.

The memory <NUM> is configured to store program code that can be executed by the processor <NUM> to perform the method described herein. Alternatively or in addition, one or more memories <NUM> may be external to (i.e. separate to or remote from) the apparatus <NUM>. For example, one or more memories <NUM> may be part of another device. Memory <NUM> can be used to store the sequence of ultrasound image data and/or any other information or data received, calculated or determined by the processor <NUM> of the apparatus <NUM> or from any interfaces, memories or devices that are external to the apparatus <NUM>. The processor <NUM> may be configured to control the memory <NUM> to store the sequence of ultrasound image data and/or the any other information or data received, calculated or determined by the processor.

In some embodiments, the memory <NUM> may comprise a plurality of sub-memories, each sub-memory being capable of storing a piece of instruction data. For example, at least one sub-memory may store instruction data representing at least one instruction of the set of instructions, while at least one other sub-memory may store instruction data representing at least one other instruction of the set of instructions.

It will be appreciated that <FIG> only shows the components required to illustrate this aspect of the disclosure and, in a practical implementation, the apparatus <NUM> may comprise additional components to those shown. For example, the apparatus <NUM> may further comprise a display. A display may comprise, for example, a computer screen, and/or a screen on a mobile phone or tablet. The apparatus may further comprise a user input device, such as a keyboard, mouse or other input device that enables a user to interact with the apparatus, for example, to provide initial input parameters to be used in the method described herein. The apparatus <NUM> may comprise a battery or other power supply for powering the apparatus <NUM> or means for connecting the apparatus <NUM> to a mains power supply.

In some embodiments, the apparatus <NUM> may be comprised in an ultrasound system. For example, an ultrasound system may comprise the apparatus <NUM> described above, and further comprise a transducer with which to record the sequence of ultrasound image data.

<FIG> shows an example ultrasound system <NUM>. The ultrasound system <NUM> may comprise the apparatus <NUM> described above. In other embodiments, components of the ultrasound system <NUM> may be adapted to perform the method <NUM> described below.

The system <NUM> comprises an array transducer probe <NUM> which has a transducer array <NUM> for transmitting ultrasound waves and receiving echo information. The transducer array <NUM> may comprise CMUT transducers; piezoelectric transducers, formed of materials such as PZT or PVDF; or any other suitable transducer technology. In this example, the transducer array <NUM> is a two-dimensional array of transducers <NUM> capable of scanning either a 2D plane or a three dimensional volume of a region of interest. In another example, the transducer array may be a 1D array.

The transducer array <NUM> may be coupled to a microbeamformer <NUM> which controls reception of signals by the transducer elements. Microbeamformers are capable of at least partial beamforming of the signals received by sub-arrays, generally referred to as "groups" or "patches", of transducers as described in <CIT>), <CIT>), and <CIT>).

In an alternative embodiment, instead of a microbeamformer <NUM>, the transducer array may be operated directly by a main system beamformer (not shown in <FIG>).

The system <NUM> may further comprise a transmit/receive (T/R) switch <NUM>, which the microbeamformer <NUM> can be coupled to and which switches the array between transmission and reception modes.

The transmission of ultrasound beams from the transducer array <NUM> is directed by a transducer controller <NUM> coupled to the microbeamformer by the T/R switch <NUM> and a main transmission beamformer (not shown), which can receive input from the user's operation of the user interface or control panel <NUM>. The controller <NUM> can include transmission circuitry arranged to drive the transducer elements of the array <NUM> (either directly or via a microbeamformer) during the transmission mode.

It is noted that in an alternative embodiment, where instead of a microbeamformer <NUM>, the transducer array is operated directly by a main system beamformer, a T/R switch <NUM> may protect the main beamformer <NUM> from high energy transmit signals.

In a typical line-by-line imaging sequence, the beamforming system within the probe may operate as follows. During transmission, the beamformer (which may be the microbeamformer or the main system beamformer depending upon the implementation) activates the transducer array, or a sub-aperture of the transducer array. The sub-aperture may be a one dimensional line of transducers or a two dimensional patch of transducers within the larger array. In transmit mode, the focusing and steering of the ultrasound beam generated by the array, or a sub-aperture of the array, are controlled as described below.

For each line (or sub-aperture), the total received signal, used to form an associated line of the final ultrasound image, will be a sum of the voltage signals measured by the transducer elements of the given sub-aperture during the receive period. The resulting line signals, following the beamforming process below, are typically referred to as radio frequency (RF) data. Each line signal (RF data set) generated by the various sub-apertures then undergoes additional processing to generate the lines of the final ultrasound image. The change in amplitude of the line signal with time will contribute to the change in brightness of the ultrasound image with depth, wherein a high amplitude peak will correspond to a bright pixel (or collection of pixels) in the final image. A peak appearing near the beginning of the line signal will represent an echo from a shallow structure, whereas peaks appearing progressively later in the line signal represent echoes from structures at increasing depths within the subject.

In addition, upon receiving the echo signals from within the subject, it is possible to perform the inverse of the above described process in order to perform receive focusing. In other words, the incoming signals may be received by the transducer elements and subject to an electronic time delay before being passed into the system for signal processing. The simplest example of this is referred to as delay-and-sum beamforming. It is possible to dynamically adjust the receive focusing of the transducer array as a function of time.

The structural and motion signals produced by the B mode and Doppler processors are coupled to a scan converter <NUM> and a multi-planar reformatter <NUM>. The scan converter <NUM> arranges the echo signals in the spatial relationship from which they were received in a desired image format. In other words, the scan converter acts to convert the RF data from a cylindrical coordinate system to a Cartesian coordinate system appropriate for displaying an ultrasound image on an image display <NUM>. In the case of B mode imaging, the brightness of pixel at a given coordinate is proportional to the amplitude of the RF signal received from that location. For instance, the scan converter may arrange the echo signal into a two dimensional (2D) sector-shaped format, or a pyramidal three dimensional (3D) image. The scan converter can overlay a B mode structural image with colors corresponding to motion at points in the image field, where the Doppler-estimated velocities to produce a given color. The combined B mode structural image and color Doppler image is able to depict tissue motion and blood flow within the structural image field. The multi-planar reformatter will convert echoes that are received from points in a common plane in a volumetric region of the body into an ultrasound image of that plane, as described in <CIT>). A volume renderer <NUM> converts the echo signals of a 3D data set into a projected 3D image as viewed from a given reference point as described in <CIT>).

The 2D or 3D images are coupled from the scan converter <NUM>, multi-planar reformatter <NUM>, and volume renderer <NUM> to an image processor <NUM> for further enhancement, buffering and temporary storage for display on an image display <NUM>. The imaging processor may be adapted to remove certain imaging artifacts from the final ultrasound image, such as for example,: acoustic shadowing, for example caused by a strong attenuator or refraction; posterior enhancement, for example caused by a weak attenuator; reverberation artifacts, for example where highly reflective tissue interfaces are located in close proximity; and so on. In addition, the image processor may be adapted to handle certain speckle reduction functions, in order to improve the contrast of the final ultrasound image.

In addition to being used for imaging, the blood flow values produced by the Doppler processor <NUM> and tissue structure information produced by the B mode processor <NUM> are coupled to a quantification processor <NUM>. The quantification processor may be used for making measurements in the images. The quantification processor may receive input from a user control panel <NUM>, such as the seed points for locating the surface of the muscle, as described in detail below.

Output data from the quantification processor is coupled to a graphics processor <NUM> for the reproduction of measurement graphics and values with the image on the display <NUM>, and for audio output from the display device <NUM>. The graphics processor <NUM> can also generate graphic overlays for display with the ultrasound images. These graphic overlays can contain standard identifying information such as patient name, date and time of the image, imaging parameters, and the like. For these purposes the graphics processor receives input from the user interface <NUM>, such as patient name. The user interface is also coupled to the transmit controller <NUM> to control the generation of ultrasound signals from the transducer array <NUM> and hence the images produced by the transducer array and the ultrasound system. The transmit control function of the controller <NUM> is only one of the functions performed. The controller <NUM> also takes account of the mode of operation (given by the user) and the corresponding required transmitter configuration and band-pass configuration in the receiver analog to digital converter. The controller <NUM> can be a state machine with fixed states.

The skilled person will appreciate that the detail provided above and the components illustrated in <FIG> are an example only and that an ultrasound system may have different components to those illustrated therein.

Turning now back to <FIG>, and the functionality of the processor <NUM>, as noted above, the processor <NUM> is caused to, responsive to receiving input indicating a point in an image corresponding to an anatomical feature or sub-feature, determine if sufficient points to perform a measurement have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement; and if sufficient points to perform a measurement have been received, perform the measurement.

Turning to <FIG>, there is a computer implemented method <NUM> for use in processing image data relating to a pelvic floor of a subject. Embodiments of the method <NUM> may be performed, for example by an apparatus such as the apparatus <NUM> described above.

Briefly, in a first step <NUM>, the method <NUM> comprises, responsive to receiving input indicating a point in an image corresponding to an anatomical feature or sub-feature, determining if sufficient points to perform a measurement have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement. In a second step <NUM> the method comprises, if sufficient points to perform a measurement have been received, performing the measurement.

The pre-stored anatomical data may be a data set which comprises a list of anatomical features and/or sub-features, where groups of features from the list may be usable, or required, to perform particular measurements. The method may therefore determine when all features and/or sub-features in a particular group corresponding to a measurement have been indicated. The data set may comprise some or all of the information illustrated in table <NUM>. It will be appreciated that the same anatomical feature or sub-feature may be used to perform different measurements, e.g. the same anatomical feature or sub-feature may be used to perform more than one measurement. The determining may involve comparing any received points (e.g. any points that have been indicated by the user for the same image data to the pre-stored anatomical data to determine if the points which have been indicated can be used to perform a measurement). Thus, received points may be stored.

A point may be determined as corresponding to an anatomical feature or sub feature based on received input from a user indicating the anatomical feature or sub feature to which the point corresponds. For example, after indicating a point, a user may select a corresponding anatomical feature or sub feature, e.g. from a drop down menu or the like. Alternatively, the anatomical feature or sub-feature to which the point corresponds may be determined by classifying the point into a class corresponding to the anatomical feature or sub feature, for example using an algorithm.

The method may further comprise determining to which anatomical feature or sub feature the point corresponds by classifying the point into a class corresponding to an anatomical feature using a first portion of the image around the point and using the position of the point relative to the image, and if the class into which the point is classified comprises sub-classes corresponding to sub-features of the anatomical feature, classify the point into a sub-class of the class using a second portion of the image around the point.

The method may be for use in processing image data relating to a pelvic floor of a subject. In particular the image data may comprise a plane of a user, such as a sagittal plane (e.g. a mid-sagittal plane), a coronal plane, an axial plane, (or any off-axis plane) for example. The anatomical features or sub features may usable to assess the pelvic floor of a subject. Where the method is used for assessing pelvic floor, the anatomical feature may be at least one of: symphysis pubis, bladder region, uterus, anorectal region. A point may be determined as corresponding to one of these anatomical features. However, some of these anatomical features have sub-features which may also be usable for assessing the pelvic floor of a subject. Therefore, it may be advantageous to identify a sub-feature of an anatomical feature where an anatomical feature comprises a sub-feature. For example, for a bladder region, there are several points of interest, such as the bladder neck, the urethral distal end and the bladder wall. Therefore, it may be considered that the bladder region is a general anatomical feature (corresponding to a class) and the bladder neck and the bladder wall are sub-features (corresponding to sub-classes) of the bladder region. In an example, the user may identify the feature or sub-feature to which the point corresponds. In a further example, the method may classify the feature or sub-feature.

Where the method comprises classification of a point, where a point is determined as being within the bladder region in the method, the method may therefore go on to identify the point as corresponding to a further sub-feature of the bladder. The sub-anatomical features may be at least one of: bladder neck; urethral distal end; point on bladder wall; rectal ampulla; point on anal canal; point on rectum; uterus neck; first bladder long axis end; second bladder long axis end; first bladder short axis end; second bladder short axis end; first detrusor wall measurement line <NUM> end; second detrusor wall measurement line <NUM> end; first detrusor wall measurement line <NUM> end; second detrusor wall measurement line <NUM> end; first detrusor wall measurement line <NUM> end; second detrusor wall measurement line <NUM> end; urethra; posteria wall of bladder; detrusor wall upper border; detrusor wall lower border; first bladder side; second bladder side; third bladder side; fourth bladder side; anal canal, and so on.

It will be appreciated that the method is not limited to the detection or identification of these anatomical features or sub features, and may be applied to detect or identify any relevant features. These anatomical features or sub features may be usable in order for the method to perform measurements based on the position of the point or points in the image. The measurement may be a measurement usable for assessing the pelvic floor of a subject.

The received input may be input from a user interacting with an interface, and may be, for example, a click on a point in an image corresponding to an anatomical feature. It will be appreciated that the method is not limited herein to a two dimensional (2D) image or image data, and could instead be applied to three dimensional (3D) images or image data, or 4D images or image data. Thus, the first portion of the image around the point and/or the second portion of the image around the point may be an area or may be a volume. It will be further appreciated that while a user may interact with a 2D image, the 2D image may be a representation of a 3D (or 4D) image or image data, and therefore the point indicated in the 2D representation may be used to determine a volume around the point in a 3D image corresponding to the 2D representation.

The first portion of the image may be larger than the second portion of the image. In particular, the first portion of the image may comprise a larger area or a larger volume than the second portion of the image. For example, the first portion of the image may be used to determine the general location of the indicated point, such as in relation to an organ of a subject. The second portion of the image may be used to determine a more specific location of the indicated point, in particular in relation to a component of an organ. Therefore, the first portion may be larger to determine a general location of the point, and once the general location has been determined, the second portion may be smaller in order to more specifically locate the point. It will be appreciated that a larger region will have more distinctive characteristics than a smaller region, and therefore classification of a larger region may be performed more quickly and more accurately than a smaller region. Furthermore, by first classifying the point into a region before classifying the point into a sub-region, the accuracy and speed of the second classification may be improved, as there will be a reduced number of options for the sub-region based on the region into which the point has been classified compared to the number of sub-regions that may be present in the whole image (e.g. corresponding to different anatomical features). The size of the first region may be a first predetermined size and the size of the second region may be a second predetermined size. The sizes may be proportionate to the overall size of the image, or may be pre-set sizes. In an example, the first potion may be large enough to encompass an anatomical feature, and may be in the region of <NUM>-<NUM> square pixels for an image of size <NUM>*<NUM> pixels. For the second portion, the size may depend on the classification result relating to the first portion (e.g. the size of the second portion may depend on the anatomical features that is the result of the classification of the first portion). The second portion may be in the region of <NUM>-<NUM> square pixels for an image of size <NUM>*<NUM> pixels.

The classification may be performed using a classification algorithm, or a classification model, which has been trained using portions of training images around points corresponding to at least one of the anatomical features and the anatomical sub-features, or has been developed based on an image feature extraction method (e.g. using shape, texture and so on). Any appropriate image processing methods or deep learning methods may be used in the classification of points. For example, using training data of portions of an image labelled as the relevant anatomical features or sub-features, the classification model may be trained to classify an anatomical feature or sub feature from input of the first portion of the image and/or the second portion of the image. The classification model may also take as input the position of the point relative to the image to assist in the classification. It will be appreciated that any appropriate classification model, such as a neural network, machine learning model, deep learning neural network, image feature extraction and regression, and so on, may be used in the methods described herein.

The skilled person will be familiar with neural networks, but in brief, neural networks are a type of supervised machine learning model that can be trained to predict a desired output for given input data. Neural networks are trained by providing training data comprising example input data and the corresponding "correct" or ground truth outcome that is desired. Neural networks comprise a plurality of layers of neurons, each neuron representing a mathematical operation that is applied to the input data. The output of each layer in the neural network is fed into the next layer to produce an output. For each piece of training data, weights associated with the neurons are adjusted until the optimal weightings are found that produce predictions for the training examples that reflect the corresponding ground truths.

The skilled person will appreciate that generally the trained model may comprise any model that may be trained to take as input an image product and output a classification for one or more features in an image.

The point may be stored with a label indicating the anatomical feature or sub feature to which the point corresponds. For example, the point may be stored with a label indicating the class or sub class into which it has been classified. The point may be displayed on a display showing the image or a representation of the image. In this way, a user may be able to determine which points have been previously obtained, and avoid repeating the same input (e.g. clicks on the same point). For particular points, additional features may be determined and/or displayed on the display. For example, if the point is identified by a user as, or is classified into a class, corresponding to the symphysis pubis (SP), the method may cause a horizontal line through the point to be determined, and may displayed the line on a display showing the image.

The method may comprise performing a measurement. For example, a measurement is performed if sufficient points have been received to perform the measurement. In particular, the measurement may be performed using the position of the point in the image. For example, several anatomical features may be required to perform a measurement. Each time input is received indicating a point in the image, the point may also be indicated by a user as corresponding to an anatomical feature or sub feature, and when enough anatomical features have been identified that a measurement can be performed, the method may further comprise performing the measurement. For pelvic floor images, the measurement may be at least one of: bladder to symphysis pubis; uterus to symphysis pubis, rectal ampulla to symphysis pubis; urethral tilt angle; retrovesical angle; anorectal angle; bladder long axis; bladder short axis; first detrusor wall thickness; second detrusor wall thickness; third detrusor wall thickness; hiatus area; hiatus long axis; hiatus short axis; first levator ani thickness; second levator ani thickness; third levator ani thickness; left levator-urethral gap; right levator urethral gap; first anal sphincter defect angle; second anal sphincter defect angle; third anal sphincter defect angle; fourth anal sphincter defect angle; fifth anal sphincter defect angle. It will be appreciated that any relevant measurements may be taken using located anatomical or sub anatomical features in an image of a subject.

The method may further comprise performing a measurement relating to the anatomical feature or sub feature corresponding to the class or sub-class into which the point is classified, and wherein the measurement is performed using the position of the point in the image. The method may further comprise classifying a plurality of points into classes and sub-classes, and perform the measurement once points have been identified as belonging to classes and/or sub-classes corresponding to all the anatomical features and/or sub-features required for the measurement to be performed. For example, several anatomical features may be required to perform a measurement. Each time input is received indicating a point in the image, the point may be classed as belonging to a particular class or sub class corresponding to an anatomical feature or sub feature, and when enough anatomical features have been identified that mean a measurement can be performed, the method may further comprise performing the measurement. For pelvic floor images, the measurement may be at least one of: bladder to symphysis pubis; uterus to symphysis pubis, rectal ampulla to symphysis pubis; urethral tilt angle; retrovesical angle; anorectal angle; bladder long axis; bladder short axis; first detrusor wall thickness; second detrusor wall thickness; third detrusor wall thickness; hiatus area; hiatus long axis; hiatus short axis; first levator ani thickness; second levator ani thickness; third levator ani thickness; left levator-urethral gap; right levator urethral gap; first anal sphincter defect angle; second anal sphincter defect angle; third anal sphincter defect angle; fourth anal sphincter defect angle; fifth anal sphincter defect angle. It will be appreciated that any relevant measurements may be taken using located anatomical or sub anatomical features in an image of a subject.

Once a measurement has been performed, the result of the measurement may be stored. Once measurements have been performed corresponding to all the measurements required to make a further measurement, the further measurement may be performed. For example, a calculation may be performed using at least one measurement. The measurements and/or further measurements may be stored.

The identified anatomical features (e.g. the class or sub class the point is classified into, or is identified by a user) may be stored in a memory or register. It will be appreciated that it is not necessary for a user to repeatedly indicate an anatomical feature that is usable to perform more than one measurement. For example, once an anatomical feature has been located, it may be used in any measurement that requires that anatomical feature. This is advantageous as it does not require a user repeatedly inputting the same point, and therefore improves efficiency of processing image data.

The classification of the point may be further based on the classification of a preceding point. For example, if the method has classified a point which has been previously indicated, it may use this classification to determine the classification of a current point. In particular, a user indicating a sequence of points may do so based on a predetermined sequence. For example, to measure the retrovesical angle, a user will typically click on the urethra, followed by the bladder neck, and then the posterior wall of the bladder. Therefore, the classification may expect points to follow a particular sequence of input of points. The method may therefore use the location of the point in an order of input of points relative to a predicted sequence of input of points in order to classify a current point. Furthermore, the position of the point in the image relative to a previously classified point may be used to determine the classification of a current point. For example, two components of the same organ may typically have a similar relative position to one another even in different subjects or images. Therefore, where a currently indicated point is positioned relative to a previously indicated point at a location where an anatomical feature is expected based on the anatomical feature corresponding to the classification of the previously classified point, the classification may also be based on whether the anatomical feature is expected to be at the location.

The confidence of the classification of a previously classified point may be updated based on the classification of the point. For example, if a current point is classified into the same class (or sub class) as a previously classified point, the confidence of the previously classified point may be reduced. If a current point is classified into a different class (or sub class) to any previously classified point, the confidence of the previously classified point may be increased.

The confidence of the classification of a previously classified point may be updated based on the position of the point in the image relative to the previously classified point. For example, where a previously classified point (e.g. corresponding to an anatomical feature) is expected to be located relative to another point corresponding to another anatomical feature, classification of a current point may be used to determine if the previously classified point has likely been correctly identified. If it is determined that the confidence of the previously classified point has likely been correctly identified based on the classification and location of the current point, the confidence of the classification of the previous point may be increased. Similarly, if it is determined that the confidence of the previously classified point has likely been incorrectly identified based on the classification and location of the current point, the confidence of the classification of the previous point may be decreased.

The method may further comprise, if the point is classified into the same class, or the same sub-class where the class is separable into sub-classes, as a previously classified point, requesting input indicating which classification should be used for at least one of the point and the previously classified point. The method may further comprise, responsive to receiving an indication of a classification to be used for at least one of the point or the previously classified point, updating the classification of at least one of the point or the previously classified point based on the received indication. Thus, the accuracy of classification can be improved by the confirmation or alteration of a classification.

In an example, ultrasound images may be obtained by the systems described above. For example, a user can use the ultrasound system with either a linear probe or a volume probe to acquire 2D images of the mid-sagittal plane at the rest state of a subject. The user may then indicate a point in the image by clicking on a point in the image. In an example of pelvic floor assessment, the click may be any point of the <NUM> points needed as the input to perform one or more of the <NUM> measurements (the required points and measurements are outlined in table <NUM>). The method may then comprise determining which of the <NUM> landmarks (anatomical features or sub features) the user has clicked. As discussed above, the method may use a two-step classification using two differently sized portions of the images around the clicked points. In addition, the position of the clicked point within the image may also be used for the classification. Alternatively, a user may indicate which of the <NUM> landmarks the user has clicked. <FIG> illustrates examples of ultrasound images relating to a pelvic floor of a subject (in particular a mid sagittal plane) where points corresponding to anatomical features or sub features are indicated by dots (points). In this example, points corresponding to SP, bladder neck, uterus, and rectal ampulla have been identified. These points may have been indicated (e.g. by a user). As is illustrated in this Figure, these points may be used in order to perform measurements.

<FIG> illustrates an example of an ultrasound image. In an existing system, to measure the distance from the bladder neck (B) <NUM> to a horizontal reference line <NUM> through the symphysis pubis (SP) <NUM>, the user is required to identify and click on two points to create the horizontal reference line <NUM> through the SP (the horizontal reference line through the SP is an important reference line for judging organ prolapse), and to identify and click on two further points to create a vertical line to take the distance measurement from the bladder neck <NUM> to the horizontal reference line <NUM> through the SP <NUM> - one on the bladder neck <NUM> and one on the horizontal reference line <NUM>. The user is also required to indicate which points have been clicked to a computing device, and indicate which measurement is to be performed. A similar process may be used to determine the distance <NUM> from the uterus (UT) <NUM> to the horizontal reference line <NUM>, and the distance <NUM> from the rectal ampulla (RA) <NUM> to the horizontal reference line <NUM>.

However, the methods described herein may instead automatically perform a measurement once sufficient anatomical features or sub features have been identified for that measurement to be performed. For example, there may be provided a pre-stored set of anatomical features or sub-features, where groups of features may be usable in order to perform a measurement. Therefore, by comparison of the identified features or sub features with the pre-stored set of features, it can be determined if sufficient features or sub features have been identified/indicated to perform a measurement, and the measurement may be performed once the features required for a measurement have been identified. Furthermore, the method may automatically classify a point using a portion of the image around the point.

Once a point has been identified, a measurement may be performed and may be displayed on the image currently being viewed by a user. As an example, once the SP has been identified, a horizontal line passing through the SP may be determined. Once a point corresponding to one of the bladder neck, the uterus, or the rectal ampulla, has been identified the method may comprise automatically performing the measurements outlined above between each of these features and the horizontal line passing through the SP. The distance between one of the bladder neck, the uterus or the rectal ampulla, and the SP, may be shown on the image.

There are two further measurements including the urethral tilt angle <NUM> (shown in <FIG> and the retrovesical angle <NUM> (shown in <FIG>) that also use the bladder neck landmark <NUM> which was obtained in relation to <FIG>. In order to obtain these further measurements, in current methods, it may be necessary for a user to click again on the same anatomical landmark to take a measurement, which is the bladder neck <NUM> in this case, in addition to the urethral distal end <NUM> for the urethral tilt angle <NUM>, and the urethral distal end <NUM> and a point on the bladder wall <NUM> for the retrovesical angle <NUM>. However, using the methods described herein, once an anatomical feature has been identified, there is no need for a user to re-find the same feature to perform a different measurement. In this example, that means that once the bladder neck <NUM> and urethral distal end <NUM> have been identified, the urethral tilt angle <NUM> may be determined, and only one further click to identify a point on the bladder wall is required in order that the points required for the retrovesical angle measurement to be performed are obtained.

<FIG> illustrates a flow chart of the method according to an example in which classification is performed. As can be seen in this Figure, the method may comprise receiving an image <NUM>, and receiving input indicating a point in the image <NUM>, for example, a point clicked by a user. A first portion of the image may be obtained, for example by cropping the image around the point <NUM>. A first classification stage may then classify the point into a class corresponding to an anatomical feature using a first portion of the image around the point and using the position of the point relative to the image <NUM>, where the first portion of the image is relatively large. The point may therefore be classified into a coarse class <NUM>. For example, where the image is related to pelvic floor assessment, the coarse class may be divided into the four classes based on the distinctive location of the anatomical features relating to the class in the image and the distinctive echo presentation in these regions. In particular, the classes in this example are SP <NUM>, bladder region <NUM>, uterus <NUM> and anorectal region <NUM>. Once the region has been narrowed down, a second classification may be performed <NUM> for anatomical features which may be divisible into sub features. In this example, the bladder region and the anorectal region are separable into a number of smaller anatomical sub-features. In particular, the bladder region may be divisible into sub-classes of urethra, bladder neck, posteria wall of bladder, detrusor wall upper border, detrusor wall lower border, four sides of the bladder <NUM>. The anorectal region may be divisible into sub regions of the anal canal, analrectum ampulla, and rectum <NUM>. A second portion, e.g. a smaller ROI/detection anchor size, may be used to separate the landmarks that are crowded in nearby locations by classifying the point using the second portion. Once a class or sub-class has been determined for a point, the click order and related clicks <NUM> may be used in order to improve the confidence of a classification. For example, it may be assumed that the user will click the landmarks (e.g. the anatomical features or sub-features) in a rational order, where this information can then be used to update the classification results and boost the confidence of the classification. For example, to measure the retrovesical angle, a user will typically click on the urethra, followed by the bladder neck, and then the posterior wall of the bladder. The click on the posterior wall of the bladder may be mistaken as one of the points on the bladder long axis for measuring the residual bladder volume. If the click order and prior click classification information is used, the method may reduce the chance of mistaking similar and close-by landmarks. The landmark result may therefore be updated <NUM>.

Once the classification has been performed, the method may further comprise enabling the user to correct the classification result if needed. The user may use a user interface such as a control panel or a mouse to access a drop-down menu of classification results listed in the order of classification likelihood. The classified result of the point may be registered in an internal processor/memory. If the user clicks on a point that highly coincidence with a prior point and is therefore classified into the same class or sub class, another function may be triggered to enable the user to confirm whether the prior classified landmark needs to be updated.

A processor may be continuously searching for identified features or sub features so that once all the required inputs for a measurement are completed, the measurement will be automatically computed and displayed on the image. If insufficient input is provided, then the processor may wait for the next input until enough features or sub features are identified for a measurement.

This process is illustrated in the <FIG>. For example, in <FIG> input corresponding to clicks on points in the image is received (shown as crosses in the images). Using the methods described herein, each point is either indicated by a user as corresponding to a feature or sub feature, or classified. In this example, the two points marked by an X in the image correspond to the SP <NUM> and the bladder neck <NUM>. The positions corresponding to these features are stored along with the features. The two points of SP and bladder neck are used to determine the first bladder distance. A further point of the bladder posterior wall <NUM> is identified based on a point clicked by a user illustrated in <FIG>. However, these three points are not sufficient to perform any further measurement, and therefore no further measurement is performed. <FIG> illustrates a further point clicked by a user which is identified as the urethra <NUM>. With this additionally identified point, it becomes possible to perform measurements of a urethral incline angle as well as a retrovesical angle using the identified points. These two further measurements are therefore performed using the points.

<FIG> illustrates an example of an ultrasound image of a mid-sagittal plane of a subject with markings indicating the location of anatomical features or sub-features which may be identified using the methods described herein. Furthermore, this Figure illustrates an example of measurements which may be performed using the located anatomical features once the relevant anatomical features required to perform a measurement have been located.

In particular, this Figure illustrates anatomical features such as the SP: symphysis pubis <NUM>, B: bladder <NUM>, UT: uterus <NUM>, RA: rectal ampulla <NUM>. Further features and sub features are illustrated as the additional dots in the image (corresponding to the eighteen points required to make the eleven measurements in relation to an image of the mid-sagittal plane at rest). Measurements are illustrated as dotted lines. Measurements shown in this Figure include the vertical distance from bladder/uterus/rectal ampulla to symphysis pubis <NUM>, <NUM>, <NUM>, angle measurements including UTA: urethra tilt angle <NUM>, RVA: retrovesical angle <NUM>, ARA: anorectal angle <NUM>, the bladder long and short axes, and <NUM> bladder wall thickness measurements.

The method may further comprise performing a further measurement using a previously determined measurement. For example, once a measurement of the long and short axes for the bladder has been performed, the residual bladder volume may be calculated as being Residual Bladder Volume (ml) = Bladder long axis (cm) * Bladder short axis (cm) * <NUM>.

In another embodiment, there is provided a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method or methods described herein.

Thus, it will be appreciated that the disclosure also applies to computer programs, particularly computer programs on or in a carrier, adapted to put embodiments into practice. The program may be in the form of a source code, an object code, a code intermediate source and an object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the embodiments described herein.

Claim 1:
An apparatus for use in processing image data relating to a subject, the apparatus comprising:
a memory (<NUM>) comprising instruction data representing a set of instructions; and
a processor (<NUM>) configured to communicate with the memory and to execute the set of instructions (<NUM>), chacterized in that the set of instructions, when executed by the processor, cause the processor to:
responsive to receiving input indicating a point (<NUM>, <NUM>, <NUM>, <NUM>) in an image corresponding to an anatomical feature or sub-feature of the subject,
determine (<NUM>) if sufficient points to perform a measurement (<NUM>, <NUM>) have been received based on a comparison of anatomical features and/or sub-features corresponding to the indicated point and any other indicated points with pre-stored anatomical data, the pre-stored anatomical data indicating for at least one measurement the anatomical features and/or sub-features required to perform the at least one measurement; and
if sufficient points to perform a measurement have been received, perform (<NUM>) the measurement.