System and method for processing multimodal images

Various aspects of a system and a method to process multimodal images are disclosed herein. In accordance with an embodiment, the system includes an image-processing device that generates a structured point cloud, which represents edge points of an anatomical portion. The structured point cloud is generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. Diffusion filtering is performed to dilate edge points that correspond to the structured point cloud to mutually connect the edge points on the structured point cloud. A mask is created for the anatomical portion based on the diffusion filtering.

REFERENCE

FIELD

Various embodiments of the disclosure relate to processing of multimodal images. More specifically, various embodiments of the disclosure relate to processing of multimodal images associated with an anatomical portion of a subject.

BACKGROUND

Advancements in the field of medical imaging techniques and associated sensors or devices have made possible to visualize the interior of a body for clinical analysis and medical purposes. Different modalities, such a Computerized Tomography (CT) scanner and Magnetic Resonance Imaging (MRI) machines, provide different types of medical images for an anatomical portion-of-interest. Such different types of images are referred to as multimodal images. Multimodal images of the same anatomical portion, such as a skull portion, of the same subject may provide different visual representations and varied information depending on the modality used. It may be difficult to register such multimodal images because of different characteristics, such as structural, resolution, and/or clinical usage differences of the different imaging sensors. The multimodal images also have at least some common information content, which if located and computed accurately, registration may be achieved even for the multimodal images obtained from different sensors. Thus, an advanced technique and/or system may be required to process such multimodal images to generate enhanced visualization of one or more anatomical portions of a particular subject with improved accuracy. Such enhanced visualization may be employed by users, such as a physician, for diagnostic purposes and/or for provision of assistance in surgery.

SUMMARY

A method and a system are provided to process multimodal images substantially as shown in, and/or described in connection with, at least one of the figures, as set forth more completely in the claims.

DETAILED DESCRIPTION

The following described implementations may be found in the disclosed system and method to process multimodal images. Exemplary aspects of the disclosure may include generation of a structured point cloud by an image-processing device that represents edge points of an anatomical portion. The structured point cloud may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. Diffusion filtering may be performed to dilate edge points that correspond to the structured point cloud to mutually connect the edge points on the structured point cloud. A mask may be created for the anatomical portion from the diffusion filtering.

In accordance with an embodiment, the anatomical portion may correspond to a skull portion, a knee cap portion, or other anatomical portions of a subject. The multimodal images may be received from a plurality of medical imaging devices. The received multimodal images may correspond to different sets of unregistered images associated with the anatomical portion of a subject. The plurality of multimodal images may correspond to X-ray computed tomography (CT), magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), fluid-attenuated inversion recovery (FLAIR), and/or positron emission tomography (PET).

In accordance with an embodiment, volumetric edges of the anatomical portion of the subject may be detected by use of a first set of images. The first set of images may be obtained from at least one of the plurality of medical imaging devices that captures the anatomical portion from different points-of-view.

In accordance with an embodiment, one or more surface layers of the anatomical portion may be computed based on registration of the multimodal images. Mutual information may be computed for structures that overlap in the associated multimodal images, the anatomical portion of the subject. The amount of co-occurrence information may be measured for the overlapped structures that contain smooth gradients in the computed one or more surface layers to compute the mutual information. In accordance with an embodiment, the computed mutual information may be optimized by use of a gradient descent technique, known in the art.

In accordance with an embodiment, the computed mutual information may be modified by application of higher spatial weights around one of the computed one or more surface layers in comparison to other surface layers. The one surface layer may correspond to a skull surface. In accordance with an embodiment, skull structure information associated with the one surface layer may be identified from MRI data, based on the created mask.

In accordance with an embodiment, a plurality of multi-dimensional graphical views of the anatomical portion may be generated. The generated plurality of multi-dimensional graphical views may comprise a first set of views that further comprises the identified skull structure information associated with the one surface layer. The generated plurality of multi-dimensional graphical views may further comprise a second set of views that further comprises the identified skull structure information, together with underlying tissue information, which corresponds to the other surface layers. In accordance with an embodiment, the generated plurality of multi-dimensional graphical views may correspond to one or more perspectives of a three-dimensional (3D) view of the anatomical portion.

In accordance with an exemplary aspect of the disclosure, a structured point cloud that represents edge points of a skull portion may be generated. The structured point cloud for the skull portion may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the skull portion. Mutual information may be computed for a plurality of structures that overlap in the multimodal images associated with the skull portion. The boundary of the skull portion corresponds to one of the plurality of overlapped structures. The computed mutual information may be computed by application of higher spatial weights around a skull surface layer of the skull portion in comparison to other underlying brain surface layers of the skull portion. The skull surface layer and the underlying brain surface layers of the skull portion may be computed based on alignment of bone structure of the skull portion in the multimodal images.

FIG. 1is a block diagram that illustrates a network environment to process multimodal images, in accordance with an embodiment of the disclosure. With reference toFIG. 1, there is shown an exemplary network environment100. The network environment100may include an image-processing device102, a plurality of medical imaging devices104, multimodal images106, a server108, a communication network110, one or more users, such as a human subject112, and a medical assistant114. The multimodal images106may include different sets of unregistered images106ato106eof an anatomical portion of a subject, such as the human subject112. The image-processing device102may be communicatively coupled to the plurality of medical imaging devices104and the server108, via the communication network110.

The image-processing device102may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to process the multimodal images106, obtained from the plurality of medical-imaging devices104. In accordance with an embodiment, the image-processing device102may be configured to display a plurality of multi-dimensional, such as two-dimensional (2D) or three-dimensional (3D), graphical views of the anatomical portion. The plurality of multi-dimensional graphical views of the anatomical portion, such as the skull portion, may be a result of processing of the multimodal images106. In accordance with an embodiment, such display may occur in real-time, or near real-time, while a surgical or diagnostic procedure is performed on the anatomical region of the subject, such as the human subject112. In accordance with an embodiment, such display may also occur in preoperative, intraoperative, or postoperative states of the subject, as per user-defined configuration settings. Examples of the image-processing device102may include, but are not limited to, a user terminal or an electronic device associated with a computer-assisted surgical system or a robot-assisted surgical system, a medical device, an electronic surgical instrument, a tablet computer, a laptop, a display device, and/or a computing device.

The plurality of medical-imaging devices104may correspond to diagnostic equipment used to create visual representations of internal structures or anatomical portions of a subject, such as the human subject112. The visual representations from the diagnostic equipment may be used for clinical analysis and medical intervention. Examples of the plurality of medical-imaging devices104may include, but are not limited to, an X-ray computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a magnetic resonance angiography (MRA) scanner, a fluid-attenuated inversion recovery (FLAIR) based scanner, and/or a positron emission tomography (PET) scanner.

The multimodal images106correspond to images and/or data obtained from multimodality, such as the plurality of medical imaging devices104. For instance, the multimodal images106may include the different sets of unregistered images106ato106eof the anatomical portion, such as a skull portion, of the subject. The multimodal images106may correspond to a first set of images106aor data obtained from the MRI modality. The multimodal images106may further correspond to a second set of images106b, obtained from the CT-based medical-imaging technique. Similarly, the multimodal images106may also include a third set of images106cobtained from MRA-based medical imaging technique, a fourth set of images106dobtained from the FLAIR-based medical imaging technique, and finally, a fifth set of images106eobtained from the PET-based medical imaging technique.

The server108may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to receive and centrally store the multimodal images106and associated data obtained from the plurality of medical-imaging devices104. In accordance with an embodiment, the server108may be configured to provide the stored multimodal images106to the image-processing device102. In accordance with an embodiment, the image-processing device102may directly receive the multimodal images106from the plurality of medical-imaging devices104. In accordance with an embodiment, both the server108and the image-processing device102may be part of a computer-assisted surgical system. In accordance with an embodiment, the server108may be implemented as a plurality of cloud-based resources by use of several technologies that are well known to those skilled in the art. Examples of the server108may include, but are not limited to, a database server, a file server, an application server, a web server, and/or their combination.

The communication network110may include a medium through which the image-processing device102, the plurality of medical-imaging devices104, and/or the server108may communicate with each other. The communication network110may be a wired or wireless communication network. Examples of the communication network110may include, but are not limited to, a Local Area Network (LAN), a Wireless Local Area Network (WLAN), a cloud network, a Long Term Evolution (LTE) network, a plain old telephone service (POTS), a Metropolitan Area Network (MAN), and/or the Internet. Various devices in the network environment100may be configured to connect to the communication network110, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, infrared (IR), IEEE 802.11, 802.16, cellular communication protocols, and/or Bluetooth (BT) communication protocols.

In operation, the image-processing device102may be configured to receive the multimodal images106from the plurality of medical-imaging devices104. The received multimodal images106may correspond to the different sets of unregistered images106ato106eassociated with an anatomical portion of a subject, such as the human subject112. In accordance with an embodiment, the anatomical portion may be a skull portion of the human subject112. In accordance with an embodiment, the anatomical portion may be a knee cap part, or other anatomical portions of the human subject112. A person with ordinary skill in the art will understand that the scope of the disclosure is not limited to implementation of the disclosed system and method to process the multimodal images106of the anatomical portion of the human subject112, as shown. In accordance with an embodiment, the multimodal images106of the anatomical portion of an animal subject may be processed as required, without deviation from the scope of the disclosure.

The multimodal images106may exhibit structural, resolution, and/or clinical usage differences, in the different sets of unregistered images106ato106e. For example, structural differences may be observed when a comparison is performed among the first set of images106a, the second set of images106b, and the third set of images106c. The first set of images106a(obtained from the MRI), may provide tissue and bone structure information for an anatomical portion, such as the skull portion. The second set of images106b(obtained from the CT-based medical-imaging technique), may provide bone structure information of the anatomical portion rather than tissue information. The third set of images106cmay also comprise vessel information of the same anatomical portion, such as brain surface structures of the same subject.

In another example, the resolution of the fifth set of images106e(obtained from PET-based medical-imaging techniques), may be low as compared to other sets of images, such as the fourth set of images106d(obtained from the FLAIR). The first set of images106a(obtained from the MRI), and/or the second set of images106b(obtained from the CT-based medical-imaging technique), may have higher resolution as compared to the resolution of the fifth set of images106e. Thus, resolution differences may also be observed in the multimodal images106. Further, the first set of images106a(obtained from the MRI), may be used for the purposes of planning a surgery. On the contrary, the fourth set of images106d(obtained from the FLAIR) and the fifth set of images106e(obtained from PET) are usually used for diagnostic purposes. Thus, clinical usage differences may also be observed in the multimodal images106.

In accordance with an embodiment, to register the multimodal images106from different modalities, such as the CT and MRI, the multimodal images106must include overlapped content. The structural, resolution, and/or clinical usage differences, in the different sets of unregistered images106ato106eof the multimodal images106may make registration a difficult task. In accordance with an embodiment, the image-processing device102may be configured to locate common information content across the multimodal images106. At least a reference point, which is invariable for the same subject in two or more sets of images obtained from different modalities, may be identified and utilized for registration of the multimodal images106. For example, for registration, the image-processing device102may be configured to align the bone structure of a skull portion in the multimodal images106(which may comprise data obtained from the CT scan and the MRI of the same subject). The common information content may be identified and isolated across different image modalities as the spatial alignment of the bone structure of the skull portion, which is invariable for the same subject. Focus on a specific structure, such as the bone structure of the skull portion of the anatomy, may allow non-overlapping segments of the image content to be excluded, which increases the accuracy of the registration.

In accordance with an embodiment, the image-processing device102may be configured to detect volumetric edges of the anatomical portion of the subject, such as the human subject112. The volumetric edges of the anatomical portion may be detected by use of data obtained from at least one of the plurality of medical-imaging devices104, which captures the anatomical portion from different points-of-view. In accordance with an embodiment, the data may be a first set of images106aof the anatomical portion, such as the skull portion, obtained from the MRI.

The image-processing device102may be configured to register the multimodal images, such as the different sets of images106ato106e, based on the identified reference point. In accordance with an embodiment, the image-processing device102may be configured to compute one or more surface layers of the anatomical portion based on registration of the multimodal images. For example, the image-processing device102may compute the skull surface layer and underlying brain surface layers of the skull portion, based on the alignment of the bone structure of the skull portion in the multimodal images106.

In accordance with an embodiment, the image-processing device102may be configured to compute mutual information for overlapping structures in the multimodal images106, which may be associated with the anatomical portion of the subject. Non-overlapped structures may be considered as outliers. An amount of co-occurrence information may be measured for the overlapped structures with smooth gradients in the computed one or more surface layers. The result may be used to compute the mutual information. The mutual information for the overlapping structures in the multimodal images106may be computed by use of the following mathematical expressions:

I⁡(A,B)=∑a⁢∑b⁢PAB⁡(a,b)⁢log⁢PAB⁡(a,b)PA⁡(a)⁢PB⁡(b)(1)I⁡(A,B)=H⁡(A)+H⁡(B)-H⁡(A,B)(2)H⁡(x)=-∑i⁢p⁡(xi)⁢log⁢⁢p⁡(xi)(3)
In accordance with the expression (1), “I(A, B)” corresponds to the mutual information of two discrete random variables A and B associated with the multimodal images106. “PAB(a, b)” may be the joint probability distribution function of random variables A and B. “PA(a)” may be the marginal probability distribution function of the random variable A and “PB(b)” may be the marginal probability distribution function of the other random variable B. In accordance with expression (2), “H(A)” and “H(B)” corresponds to marginal entropies of the respective discrete random variables A and B of the associated multimodal images106, and “H(A,B)” corresponds to joint entropy of the discrete random variables A and B. In accordance with the expression (3), Shannon entropy, “H(x)” corresponds to entropy of the discrete random variable, “x”, with possible values {x1, x2, . . . , xn} for a finite sample associated with a certain number of multimodal images106, where “p(xi)” is the probability of information or character number, “i”, in the discrete random variable “x”. The Shannon entropy may measure the uncertainty in the discrete random variable “x”.

In accordance with an embodiment, the image-processing device102may be configured to modify the computed mutual information. The computed mutual information may be modified by application of higher spatial weights around one surface layer, such as skull surface layer, of the computed one or more surface layers in comparison to other surface layers.

In accordance with an embodiment, the image-processing device102may be configured to generate a structured point cloud (such as a skull point cloud), which represents edge points (such as edge points on the skull surface) of the anatomical portion. The structured point cloud may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion (described inFIG. 3Cin an example). In accordance with an embodiment, the boundary may correspond to the detected volumetric edges of the anatomical portion, such as the skull portion, of the subject.

In accordance with an embodiment, the image-processing device102may be configured to perform diffusion filtering to dilate edge points of the structured point cloud (further described inFIG. 3D). The dilation of the edge points of the structured point cloud may be performed to mutually connect the edge points in the structured point cloud. The image-processing device102may be configured to create a mask for the anatomical portion based on the diffusion filtering. The mask may be a continuous surface that may make possible optimum usage of various data, such as MRI data of the anatomical portion, to achieve accurate fusion of information obtained from the multimodality sources. The creation of the mask from the diffusion filtering may be an efficient process. The creation of the mask from the diffusion filtering may be less computationally intensive operation as compared to creation of a polygonal or triangular mesh structure from the structured point cloud to obtain a continuous surface. Further, the polygonal or triangular mesh structure may require higher storage space than the created mask.

In accordance with an embodiment, the image-processing device102may be configured to further identify skull structure information associated with the one surface layer (such as the skull surface layer), from MRI data, based on the created mask. The image-processing device102may be configured to apply the identified skull structure information from MRI data and/or the other computed and modified mutual information on and/or within the created mask to generate enhanced visual representations.

The image-processing device102may be configured to generate a plurality of multi-dimensional graphical views, such as a 3D view, of the anatomical portion as required, which may be used to plan or perform a surgery on the anatomical portion or for enhanced diagnosis of an ailment in the anatomical portion. Based on the operative state (such as preoperative, intraoperative, or postoperative), and/or received user input, different interactive graphical views of the anatomical portion may be generated. In accordance with an embodiment, user-configurations may be pre-defined or changed in real time or near real time, by the medical assistant114, based on instructions received from a registered medical practitioner. The user configurations may be used to generate different pluralities of multi-dimensional graphical views of the anatomical portion as required. Thus, the generated plurality of multi-dimensional graphical views may be user-controlled and interactive and may be changed and visualized, as medically required.

In accordance with an embodiment, the generated plurality of multi-dimensional graphical views may provide enhanced views of the anatomical portion from one or more perspectives. The generated plurality of multi-dimensional graphical views may comprise a first set of views that includes the identified skull structure information associated with the one surface layer (such as the skull surface layer). The generated plurality of multi-dimensional graphical views may also include a second set of views that includes the identified skull structure information together with underlying tissue information, which correspond to the other surface layers, such as brain surface structures when the anatomical portion is the skull portion. The brain surface structures may be gray matter, white matter, ventricular structures, vessel structure, the thalamus, and/or other tissue structures.

FIG. 2illustrates a block diagram of an exemplary image-processing device to process multimodal images, in accordance with an embodiment of the disclosure.FIG. 2is explained in conjunction with elements fromFIG. 1. With reference toFIG. 2, there is shown the image-processing device102. The image-processing device102may comprise one or more processors, such as a processor202, a memory204, one or more input/output (I/O) devices, such as an I/O device206, and a network interface208. The I/O device206may include a display210.

The processor202may be communicatively coupled to the I/O device206the memory204, and the network interface208. The network interface208may communicate with one or more servers, such as the server108, and/or the plurality of medical-imaging devices104, via the communication network110under the control of the processor202.

The processor202may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to execute a set of instructions stored in the memory204. The processor202may be further configured to process the multimodal images106received from the plurality of medical-imaging devices104or a central device, such as the server108. The processor202may be implemented based on a number of processor technologies known in the art. Examples of the processor202may be an X86-based processor, X86-64-based processor, a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a central processing unit (CPU), an Explicitly Parallel Instruction Computing (EPIC) processor, a Very Long Instruction Word (VLIW) processor, and/or other processors or circuits.

The memory204may comprise suitable logic, circuitry, and/or interfaces that may be configured to store a machine code and/or a set of instructions executable by the processor202. The memory204may be configured to store information from one or more user profiles associated with physiological data or medical history of the subject (such as the human subject112). The memory204may be further configured to store user-defined configuration settings to generate the plurality of multi-dimensional graphical views of the anatomical portion. The plurality of multi-dimensional graphical views of the anatomical portion may be displayed on a user interface (UI) rendered on the display210. The UI may be a 3D viewer or a 2D viewer. The memory204may be further configured to store operating systems and associated applications. Examples of implementation of the memory204may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or a Secure Digital (SD) card.

The I/O device206may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to receive an input from and provide an output to a user, such as the medical assistant114. The I/O device206may include various input and output devices that may be configured to facilitate communication between the image-processing device102and the user (such as the medical assistant114). Examples of the input devices may include, but are not limited to, a touch screen, a camera, a keyboard, a mouse, a joystick, a microphone, a motion sensor, a light sensor, and/or a docking station. Examples of the output devices may include, but are not limited to, the display210, a projector screen, and/or a speaker.

The network interface208may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to communicate with one or more servers, such as the server108, and/or the plurality of medical-imaging devices104, via the communication network110(as shown inFIG. 1). The network interface208may implement known technologies to support wired or wireless communication of the image-processing device102with the communication network110. The network interface208may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and/or a local buffer. The network interface208may communicate via wired or wireless communication with the communication network110. The wireless communication may use one or more of the communication standards, protocols and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, LTE, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for email, instant messaging, and/or Short Message Service (SMS).

The display210may be realized through several known technologies, such as Cathode Ray Tube (CRT) based display, Liquid Crystal Display (LCD), Light Emitting Diode (LED) based display, Organic LED display technology, Retina display technology, and/or the like. In accordance with an embodiment, the display210may be capable of receiving input from the user (such as the medical assistant114). In such a scenario, the display210may be a touch screen that enables the user to provide the input. The touch screen may correspond to at least one of a resistive touch screen, a capacitive touch screen, or a thermal touch screen. In accordance with an embodiment, the display210may receive the input through a virtual keypad, a stylus, a gesture-based input, and/or a touch-based input. In such a case, the input device may be integrated within the display210. In accordance with an embodiment, the image-processing device102may include a secondary input device apart from the display210that may be a touch screen based display.

In operation, the processor202may be configured to receive the multimodal images106from the plurality of medical-imaging devices104, by use of the network interface208. The received multimodal images106may correspond to different sets of unregistered images106ato106e, associated with the anatomical portion of the subject, such as the human subject112. The operations performed by the processor202have been further described in theFIGS. 3A to 3F, by an example of the skull portion of the human subject112, as the anatomical portion. Notwithstanding, the anatomical portion may also be a knee cap part, or other anatomical portions of the subject of which the multimodal images106may be obtained from the plurality of the medical-imaging devices104, without limiting the scope of the disclosure.

FIGS. 3A to 3F, collectively, illustrate an exemplary scenario for implementation of the disclosed system and method to process multimodal images, in accordance with an embodiment of the disclosure.FIG. 3Aillustrates receipt of multimodal images for a skull portion of a subject in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Ais explained in conjunction withFIG. 1andFIG. 2. With reference toFIG. 3A, there are shown medical images302ato302eof the same skull portion of the same subject received from the plurality of medical-imaging devices104, such as an MRI scanner304a, a CT scanner304b, an MRA scanner304c, a FLAIR scanner304d, and a PET scanner304e, respectively. There is further shown a bone structure306of the skull portion of the human subject112, common to the medical images302ato302e.

In accordance with the exemplary scenario, the medical images302ato302eof the skull portion may correspond to the multimodal images106. The medical image302amay be an output of the MRI scanner304aof the skull portion of the human subject112. A number of medical images may be obtained from the MRI scanner304afrom different points-of-view that may be referred to as a first set of medical images. The first set of medical images may correspond to first set of images106a(FIG. 1). As the medical image302arepresents a view of the skull portion from one point-of-view, the first set of medical images may represent a captured view of the skull portion from different points-of-view. Similarly, the medical image302bmay be obtained from the CT scanner304b. The medical image302cmay be obtained from the MRA scanner304c. The medical image302dmay be obtained from the FLAIR scanner304d, and finally the medical image302emay be obtained from the PET scanner304e. The output, such as the medical images302ato302e, received from multimodal sources, as described above, may be stored at a central device, such as the server108. In such a case, the processor202may receive the medical images302ato302efrom the server108. In accordance with an embodiment, the medical images302ato302emay be stored at the memory204.

In accordance with an embodiment, the processor202may be configured to process the received medical images302ato302e. The processor202may be configured to align the bone structure306of the same skull portion of the same human subject112for the registration of the unregistered medical images302ato302e. As the bone structure306is invariable for the same human subject112, it may be used as a reference point to preliminarily register the medical images302ato302e. The processor202may be configured to identify and isolate the bone structure306of the skull portion across the received medical images302ato302e. This makes possible exclusion of the non-overlapped part or outliers of the bone structure306in the medical images302ato302e.

In accordance with an embodiment, the processor202may be configured to detect volumetric edges of the skull portion of the human subject112, by use of the first set of medical images captured by the MRI scanner304afrom different points-of-view (also referred to as MRI slices). In other words, different medical images or data captured from various perspectives for the same skull portion from a single modality, such as the MRI scanner304a, may also be used to detect the volumetric edges of the skull portion based on the alignment of the bone structure306as the reference point. In accordance with an embodiment, the volumetric edges of the skull portion may represent boundary of the skull portion in a 3D space.

FIG. 3Billustrates surface layers of the skull portion computed based on the registration of the multimodal images in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Bis explained in conjunction withFIGS. 1, 2, and 3A. With reference toFIG. 3B, there is shown a skull surface layer308and a brain surface layer310, computed based on the alignment of the bone structure306of the skull portion in the medical images302ato302e. The skull surface layer308may represent the skull surface of the skull portion. The brain surface layer310may include one or more brain surface structures, such as a cerebrum surface structure, cerebellum surface structure, vessel structures, other brain tissue information, or brain ventricular structures.

In accordance with an embodiment, the processor202may be configured to compute one or more surface layers of the skull portion based on the registration. The processor202may compute the skull surface layer308, based on the alignment of the bone structure306of the skull portion in the medical images302ato302e(such as the multimodal images). In accordance with an embodiment, the processor202may compute both the skull surface layer308and the underlying brain surface layer310of the skull portion, based on the alignment of the bone structure of the skull portion in the medical images302ato302e. In accordance with an embodiment, the first set of medical images, such as MRI data, or data obtained from one or two modality instead of all of the plurality of medical-imaging devices104, may be used as required for computation of the one or more surface layers of the skull portion.

In accordance with an embodiment, the processor202may be configured to compute mutual information for structures that overlap in the medical images302ato302e, associated with the skull portion of the human subject112. The mutual information may be computed, in accordance with the mathematical expressions (1), (2), and/or (3), as described inFIG. 1. The amount of co-occurrence information may be measured for the overlapped structures with smooth gradients in the computed one or more surface layers (such as the skull surface layer308and the brain surface layer310), to compute the mutual information.

In accordance with an embodiment, the processor202may be configured to modify the computed mutual information by application of higher spatial weights around one surface layer, such as a skull surface, of the computed one or more surface layers in comparison to other surface layers. In other words, the reliable structures, such as the skull surface layer308, may be weighted more than the comparatively less reliable structures, such as vessel structures of the brain surface layer310. The application of higher spatial weights around the reliable structures increases the accuracy for computation of the mutual information across the medical images302ato302e.

FIG. 3Cillustrates creation of a mask for a skull portion in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Cis explained in conjunction withFIGS. 1, 2, 3A, and3B. With reference toFIG. 3C, there is shown a skull point cloud312and a mask314. The skull point cloud312corresponds to the structured point cloud of the anatomical portion. In accordance with an embodiment, the skull point cloud312may represent edge points of the detected volumetric edges of the skull portion, such as the boundary of skull surface, as point cloud. The mask314may be a continuous structure generated from the skull point cloud312. The mask314may represent the skull surface layer308of the skull portion. The mask314may also be representative of a current skull state, such as an open state of skull during a surgery or a closed state of skull in the preoperative or postoperative phase of a surgery.

In accordance with an embodiment, the processor202may be configured to generate the structured point cloud, such as the skull point cloud312, which represents edge points on the skull surface. The structured point cloud may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the skull portion. In accordance with an embodiment, the boundary of the skull portion may correspond to the detected volumetric edges of the skull portion of the human subject112.

In accordance with an embodiment, the unstructured point cloud may correspond to the point cloud obtained from 3D scanners or other point cloud generators known in the art, such as a laser range scanner (LRS). In accordance with an embodiment, the unstructured point cloud may correspond to the point cloud obtained by use of stereoscopic images from stereo vision, or based on computer vision that may capture the skull portion from a plurality of points-of-view. In accordance with an embodiment, the unstructured point cloud may correspond to point cloud created from the 2D medical images302ato302e(multimodal images of the skull portion).

In accordance with an embodiment, the processor202may be configured to perform diffusion filtering to dilate edge points of the skull point cloud312to mutually connect the edge points in the skull point cloud312. The processor202may be configured to create the mask314for the skull portion based on the diffusion filtering.

FIG. 3Dillustrates diffusion filtering of edge points of an exemplary skull point cloud in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Dis explained in conjunction withFIGS. 1, 2, 3A, 3B, and 3C. With reference toFIG. 3D, there is shown a skull point cloud312, a point center316, and a graph318.

The point center316corresponds to a centroid of a point of the skull point cloud312, as shown. The graph318corresponds to a diffusion filter that represents the filter strength on the Y-axis and distance from the point center316on the X-axis, as shown. The diffusion filter domain may be a 3D sphere with the same depicted profile in all three directions (such as X-, Y-, and Z-axis directions), as illustrated by the arrows.

In accordance with an embodiment, the processor202may be configured to control the thickness of the skull surface layer308. Based on the calculation of total time taken for the decay of the diffusion filter, and subsequent configuration of the total time, the thickness of the skull surface layer308may be controlled. In other words, the skull thickness may be controlled based on how fast the diffusion filter decays. In accordance with an embodiment, the diffusion filter may be centered at each point of the skull point cloud312and convolved with the skull point cloud312. Accordingly, each point of the skull point cloud312may dilate to mutually connect with each other. Such dilation and mutual connection may occur in all the three directions, such as in the X-, Y-, and Z-direction, to create the mask314of the skull portion.

In accordance with an embodiment, the processor202may be configured to identify skull structure information associated with the skull surface layer308, from the MRI data based on the created mask314. In accordance with an embodiment, the processor202may be configured to identify tissue information of the brain surface layer310, based on the computed mutual information, in accordance with the mathematical expressions (1), (2), and/or (3), as described inFIG. 1.

FIG. 3Eillustrates generation of an enhanced view of the skull portion in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Eis explained in conjunction withFIGS. 1, 2, 3A, 3B, 3C, and 3D. With reference toFIG. 3E, there is shown an enhanced view320of the skull portion.

The processor202may be configured to utilize the MRI data of the skull portion and the created mask314, to generate the enhanced view320of the skull portion. In accordance with an embodiment, the MRI data of the skull portion may be applied on the created mask314for the generation of the enhanced view320of the skull portion. The MRI data may be the identified skull structure information associated with the skull portion. In accordance with an embodiment, the modified mutual information associated with the skull surface layer308and other computed mutual information associated with the skull portion may be further utilized and applied on the created mask314, to generate the enhanced view320of the skull portion.

FIG. 3Fillustrates different views of a skull portion in the exemplary scenario for implementation of the system and method, in accordance with an embodiment of the disclosure.FIG. 3Fis explained in conjunction withFIGS. 1, 2, 3A, 3B, 3C, 3D, and 3E. With reference toFIG. 3F, there is shown a first top view322of the skull portion in the preoperative state and a second top view324of the skull portion in the intraoperative stage. There is further shown a first bottom view326of the skull point cloud312, a second bottom view328of the skull portion in the intraoperative state, and a third bottom view330of the skull portion in the preoperative state together with brain tissue information332.

The processor202may be configured to generate a plurality of multi-dimensional graphical views, such as the views322to332, of the skull portion. The generated plurality of multi-dimensional graphical views may provide enhanced views of the skull portion from one or more perspectives. The generated plurality of multi-dimensional graphical views may comprise a first set of views that includes the identified skull structure information associated with the skull surface layer308. The first top view322, the second top view324, the first bottom view326, and the second bottom view328, all correspond to the first set of views that includes the identified skull structure information associated with the skull surface layer308.

The generated plurality of multi-dimensional graphical views may also include a second set of views that includes the identified skull structure information, together with underlying tissue information that corresponds to the other surface layers, such as brain surface structures of the brain surface layer310. The third bottom view330of the skull portion in the preoperative state, together with brain tissue information332, corresponds to the second set of views that includes the identified skull structure information together with underlying tissue information.

The processor202may be configured to control display of the generated plurality of multi-dimensional graphical views, such as a 2D view or a 3D view, of the skull portion on the UI. The displayed plurality of multi-dimensional graphical views may be interactive and user-controlled, based on input received from the I/O device206. The user input may be received by use of the UI rendered on the display210, of the image-processing device102. The display of the plurality of multi-dimensional graphical views may be changed and updated in response to the received user input, such as input provided by the medical assistant114. Such enhanced visualization of the multi-dimensional graphical views of the skull portion on the UI may be utilized by users, such as a physician, for diagnostic purposes and/or for provision of real-time or near real-time assistance in a surgery.

FIG. 4illustrates a flow chart for implementation of an exemplary method to process multimodal images, in accordance with an embodiment of the disclosure. With reference toFIG. 4, there is shown a flow chart400. The flow chart400is described in conjunction withFIGS. 1, 2, and 3A to 3F. The method, in accordance with the flowchart400, may be implemented in the image-processing device102. The method starts at step402and proceeds to step404.

At step404, multimodal images106from the plurality of medical-imaging devices104may be received. The received multimodal images106may correspond to different sets of unregistered images106ato106e, associated with an anatomical portion of a subject, such as the human subject112. The anatomical portion may be a skull portion, a knee cap part, or other anatomical portions of the subject. The subject may be the human subject112or an animal subject (not shown). At step406, volumetric edges of the anatomical portion of the subject may be detected by use of a first set of images. The first set of images from the different sets of unregistered images may be obtained from at least one of the plurality of medical-imaging devices104, such as the MRI scanner, which captures the anatomical portion from different points-of-view.

At step408, the multimodal images106may be registered based on a reference point. For example, for registration, the image-processing device102may be configured to align the bone structure306of the skull portion in the multimodal images106, such as data obtained from the CT scan and the MRI. At step410, one or more surface layers of the anatomical portion may be computed based on registration of the multimodal images106, such as the medical images302ato302e. For example, the skull surface layer308and underlying brain surface layer310of the skull portion may be computed based on the alignment of the bone structure306of the skull portion in the medical images302ato302e.

At step412, mutual information may be computed for structures that overlap in the multimodal images106, associated with the anatomical portion of the subject (such as the human subject112). The mutual information may be computed, in accordance with the mathematical expressions (1), (2), and/or (3), as described inFIG. 1. The amount of co-occurrence information may be measured for the overlapped structures with smooth gradients in the computed one or more surface layers to compute the mutual information. At step414, the computed mutual information may be modified by an application of higher spatial weights around one surface layer, such as skull surface layer308, of the computed one or more surface layers in comparison to other surface layers, such as the brain surface layer310.

At step416, a structured point cloud, such as the skull point cloud312(which represents edge points, such as edge points on the skull surface), of the anatomical portion, may be generated. The structured point cloud may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. At step418, diffusion filtering may be performed to dilate edge points of the structured point cloud to mutually connect the edge points on the structured point cloud.

At step420, a mask, such as the mask314, may be created for the anatomical portion based on the diffusion filtering. At step422, skull structure information associated with the one surface layer, such as the skull surface layer308, may be identified from MRI data, based on the created mask.

At step424, skull structure information and/or modified and computed mutual information may be applied on the created mask. At step426, a plurality of multi-dimensional graphical views, such as a 3D view, of the anatomical portion may be generated. The generated plurality of multi-dimensional graphical views may provide enhanced views of the anatomical portion from one or more perspectives. The generated plurality of multi-dimensional graphical views may comprise a first set of views that includes the identified skull structure information associated with the one surface layer, such as the skull surface. The generated plurality of multi-dimensional graphical views may also include a second set of views that includes the identified skull structure information, together with underlying tissue information that corresponds to the other surface layers, such as brain surface structures. Examples of the generated plurality of multi-dimensional graphical views of the skull portion has been shown and described inFIG. 3F. Control passes to end step428.

In accordance with an embodiment of the disclosure, the system to process multimodal images may comprise the image-processing device102(FIG. 1). The image-processing device102may comprise one or more circuits, such as the processor202(FIG. 2). The processor202may be configured to generate a structured point cloud that represents edge points of an anatomical portion based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. The processor202may be further configured to perform diffusion filtering to dilate edge points that corresponds to the structured point cloud to mutually connect the edge points on the structured point cloud. The processor202may be further configured to create a mask for the anatomical portion based on the diffusion filtering.

Various embodiments of the disclosure may provide a non-transitory computer readable medium and/or storage medium, and/or a non-transitory machine readable medium and/or storage medium with a machine code stored thereon, and/or a set of instructions executable by a machine and/or a computer to process multimodal images. The set of instructions in the image-processing device102may cause the machine and/or computer to perform the steps that comprise generation of a structured point cloud that represents edge points of an anatomical portion. The structured point cloud may be generated based on shrink-wrapping of an unstructured point cloud to a boundary of the anatomical portion. Diffusion filtering may be performed to dilate edge points that correspond to the structured point cloud to mutually connect the edge points on the structured point cloud. A mask may be created for the anatomical portion based on the diffusion filtering.