Patent ID: 12229333

Any reference signs in the claims should not be construed as limiting the scope.

DETAILED DESCRIPTION

FIG.1shows an exemplary flow chart of a computer-implemented method for visualizing interactions in an extended reality (XR) scene. The method is generally referred to reference sign100.

The method100comprises a step S102of receiving a first dataset. The first dataset represents an XR scene comprising at least a technical device. The technical device may, e.g., comprise a medical device, in particular a medical imaging device, and more particularly an MRI scanner.

The method100further comprises a step S104of displaying the received S102XR scene on an XR headset or on a head-mounted display (HMD).

The method100further comprises a step S106of providing a room for a user. The user wears the XR headset or HMD for interacting with the XR scene. The XR scene is displayed on the XR headset or HMD. The room comprises a set of optical sensors. The set of optical sensors comprises at least one optical sensor at a fixed (and/or predetermined) location relative to the room, e.g., at an edge of the room. The room may be essentially empty, at least within a predetermined, e.g., rectangular, area.

The method100further comprises a step S108of detecting, via the set of optical sensors, optical sensor data of the user as a second dataset. The optical sensor data (and/or the second dataset) are detected S108while the user is interacting in the room with the XR scene. The XR scene is displayed S104on the XR headset or HMD.

The method100still further comprises a step S110of fusing the first dataset and the second dataset for generating a third dataset.

FIG.2shows an exemplary embodiment of a computing device for visualizing interactions in an XR scene. The computing device is generally referred to by reference sign200.

The computing device200comprises a first input interface202-1configured to receive a first dataset. The first dataset represents an XR scene comprising at least a technical device. The technical device may, e.g., comprise a medical device, in particular a medical imaging device, and more particularly an MRI scanner.

The computing device200further comprises a second input interface202-2configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset. The optical sensor data (and/or the second dataset) are detected while the user is interacting in a room with the XR scene. The XR scene is displayed to the user on a XR headset or HMD. The set of optical sensors comprises at least one optical sensor at a fixed (and/or predetermined) location relative to the room.

The first input interface202-1and the second input interface202-2may be comprised in a joint input interface202.

The computing device200still further comprises a computing unit206configured to fuse the first dataset and the second dataset for generating a third dataset.

Optionally, the computing device200comprises a memory208. The memory208may be configured for storing a fusing algorithm (also: fusion algorithm).

Further optionally, the computing device200comprises an output interface204. The input interfaces202-1;202-2;202and the output interface204may be comprised in an input-and-output interface210.

The computing device200may be comprised in a system, which further comprises one or more XR headsets or HMDs, a set of optical sensors, and optionally the room. The location of at least one optical sensor may be fixed (and/or predetermined) relative to the room.

The computing device200, e.g., as part of the system, may be configured for executing the method100.

The present invention can be employed for enabling (and/or improving) virtual usability engineering, e.g., via workflow analysis in fused depth-images of operating a technical device. The technical device may in particular comprise a medical device. The operating of the technical device may, e.g., comprise a virtual MRI examination.

The operating of the technical device according to embodiments of the present invention may comprise using Deep-Learning (DL) methods and augmented reality (AR) or extended reality (XR) techniques.

Digitalization and related space-saving equipment, e.g., RGB-D cameras, and/or sensors built into cellphones (also denoted as mobile phones and/or smartphones), as well as mixed-reality, virtual reality (VR), augmented reality (AR) and extended reality (XR) tools, such as headsets and related displays (e.g., HMDs), and applications have rapidly evolved over the past years.

Digitalization is influencing the technical device industry (including medical device, and particularly medical imaging device, development) significantly. One customer (e.g., a clinic, dentist's surgery, medical office, and/or any association of the former) may, e.g., request a fully autonomous scanner (also denoted as medical imaging device) which is aware of its environment (e.g., by knowing and/or understanding who is using and/or operating the scanner, and/or by knowing and/or understanding which patient interaction and/or preparation is performed) and which can support the user situation specifically (e.g., via mixed-reality techniques).

A user of a technical device may also be denoted as (in particular human) operator of the technical device, e.g., of the medical device, and/or of the scanner. Any medical imaging device may be denoted as scanner.

Today, there is no holistic solution to the described problems of conventional product (e.g., technical device) development, as there are multiple challenges (e.g., usability- and/or requirement engineering, and/or data-collection and/or data-generation) within product development. By the inventive technique, these at least two separate problems and/or topics can be perfectly combined into one approach according to computer graphics techniques, e.g., by combining the fields of CGI (Computer Generated Images, and/or Visualization) and Computer Vision (Scene Understanding, and/or Segmentation).

For usability-engineering, separate methods have been developed to record the user's behavior when working with a functional prototype and/or software module, e.g., by counting the clicks of the user, by eye tracking to detect which are the “hot spots”, which parts of the UI (and/or GUI) gets most of the user's attention and might be improved. According to the inventive technique, all of these aspects can be combined into a fully functional “Virtual Usability Lab” using AR technologies.

AR applications are hardly (and/or seldom) used during usability-/requirement engineering. Instead, AR is conventionally used (if used at all) to purely visualize static product concepts to a customer with only limited ways of interaction (e.g., only scaling and/or moving of objects). There is no fully functional AR based prototype, covering the whole workflow (e.g., comprising hardware and software) for any (e.g., medical imaging and/or treatment) modality.

First uses of AR, which are, not directly usability- and/or requirement engineering related, provide some combination of CGI and computer vision (also denoted as visualization) methods, e.g., the so called “Holoportation” by Microsoft Research.

Holoportation aims for connecting people over long distances. E.g., both users and/or operators (in the case of two users) wear AR headsets (e.g., for CGI and/or visualization) and interact in a dedicated room (e.g., the same virtual room, and/or one green screen studio per user) equipped with multiple depth cameras (e.g., RGB-D cameras) generating a color-coded, 3D depth map of the room, including the operators and/or users.

From the one or more 3D depth maps, an algorithm may segment and/or isolate the operator A (Computer Vision) and display a color-coded 3D visualization in the AR scene of operator B. The visualization may comprise a projection as a color-coded point cloud.

As both operators, A and B, use the same equipment according to some embodiment, they may see and/or meet each other virtually in the same AR environment (and/or the same virtual room). The idea, e.g., of extensions of Holoportation, goes far beyond what technologies such as Microsoft Mesh can do today and also requires dedicated equipment, but may provide a promising, new way how to fuse real and virtual environments.

None of the conventional ideas, in particular neither Holoportation nor Microsoft Mesh, is used in the field of usability- and/or requirement engineering. Moreover, the conventional technologies (e.g., including Holoportation, and/or Microsoft Mesh) suffer from one very important point, namely the true (also correct, and/or faithful) registration between real (e.g., the room the operator and/or user is in) and the virtual world (e.g., the scene as displayed in the AR device). However, the real (and/or correct, and/or faithful) registration is very important when using the data, e.g., to generate training data for a scene analysis and/or for machine Learning (ML), in particular according to embodiments of the present invention.

Conventional “projections” are only additional AR scene objects which can be moved and rotated freely. By contrast, according to the inventive technique, the AR scene contains an interaction scheme so that the user (also denoted as operator) can trigger different actions (e.g., table movement, in particular patient table movement) according to a user input and/or so that the user can operate a virtual UI, in particular a virtual GUI (e.g., a touch panel) to mimic the real-world workflow and timing. E.g., the operator may need to position a patient correctly (e.g., relative to a technical device) and enter the necessary data into an operating console of the technical device.

Self-driving cars are an example for autonomous, environment aware systems (and/or technical devices), which use real-time scene analysis, are aware, e.g., of street environment, other cars, and/or traffic lights, and decide autonomously, e.g., when to change direction, and/or when to brake. To achieve this level of automation, a massive amount of training data for all the different algorithms is conventionally needed. Data always has been an asset for autonomous cars and is also one success factor of the company Tesla as they were the first truly entering this market and collecting new real-world training data with every new car they sell.

Especially in the fields of autonomous driving and advanced robotics, the amount of training data is still a demanding topic. Therefore, companies like, e.g., NVIDIA have developed dedicated simulation frameworks (e.g., NVIDIA Isaac Sim platform for robotics, https://developer.nvidia.com/isaac-sim, and/or NVIDI Drive Sim platform for autonomous driving, https://blogs.nvidia.com/blog/2021/11/09/drive-sim-replicator-synthetic-data-generation/?ncid=soyout-833884-vt03 #cid=gtcnov21_so-yout_en-us) especially for autonomous driving and robotics. These platforms allow for simulating real world situations (e.g., different light setting and/or weather, and/or challenging environments) and generating training data for ML Algorithms using, e.g., reinforcement- and/or transfer learning. When an ML algorithm is trained with enough simulated and/or synthetic data, there is a significantly lower demand for real-world data (e.g., corresponding to transfer learning) than if the algorithm was trained on solely real-world data. Big car vendors and/or robot vendors conventionally use these kind of simulation frameworks or even have developed their own.

The new ways of generating synthetic and/or simulated training and test data for ML based algorithms is fascinating and truly a game-changer in this exiting field. Nevertheless, the frameworks conventionally strongly rely on simulating and/or modeling the environment and not on the potential user interaction (e.g., besides people walking across the street), as the human behavior is conventionally considered too hard to model. But with these limitations, the conventional frameworks cannot be used for generating synthetic test data for an autonomous MRI scanner, which needs to understand how the user is interacting with the scene (e.g., also including the patient model).

The inventive technique uses a fully interactive and functional AR (and/or XR) framework in which new product designs, customer workflows, and/or especially the interaction between software and hardware can be explored and/or presented to the customer. The inventive technique uses advanced sensor data (e.g., a 3D depth camera data, in particular an RGB-D camera) to record the user interacting with the virtual AR (and/or XR) scene. Alternatively or in addition, the inventive technique is capable of accurate registration and/or fusion of, e.g., XR scene and optical sensor, datasets (e.g., as 5D point clouds). Further alternatively or in addition, the inventive framework can be used to generate synthetic depth camera images from any arbitrary angle within the (e.g., XR) scene using the recorded 5D point cloud data. The data, and/or datasets, can be further augmented to increase the amount of generated test data and/or be used for the design and/or training of ML based algorithms for scene understanding and/or analysis.

By the inventive technique, a “Virtual Usability Lab” is developed, in which customers can experience functional product concepts at a very early development stage, while their interactions with the virtual (e.g., XR) scene may be recorded and analyzed in real time. The Virtual Usability Lab can serve at least two purposes. Firstly, getting early customer feedback in the development process, e.g., of the technical device. Secondly, generated synthetic training data for ML algorithms (e.g., for scene understanding and/or analysis) can be applied to real-world training data (e.g., once a system, and/or technical device, is developed and installed) via ML methods, e.g., comprising transfer learning.

Applications of the inventive technique comprise integration of AR (and/or XR) into the usability- and/or requirement engineering process by allowing for very early discussion of fully interactive, functional (e.g., in view of hardware and/or software) virtual prototypes with the customer. Alternatively or addition, synthetic test data (and/or test datasets) can be generated for scene understanding ML algorithms. Further alternatively or in addition, scene and/or workflow analysis (e.g., by applying the trained algorithms to the actual scene) is enabled (and/or allowed for). Alternatively or in addition, all interactions of the user with the virtual (e.g., XR) scene may be recorded and/or tracked, and/or analyzed, e.g., to find potential bottlenecks in the human operator workflow (also denoted as user workflow or customer workflow). Alternatively or in addition, techniques such as eye-tracking (e.g., supported by the MS Hololens) may be used as (e.g., additional) sensor data (and/or observation data) for advanced (e.g., real) scene analysis.

In the following, an exemplary embodiment is described.

In a first step (step1), a visualization of an XR (and/or AR) scene is provided. A fully interactive and functional AR (and/or XR) scene may be implemented using the Microsoft Hololens2 (as an example of an XR headset or HMD) for an MRI examination including a magnet and/or field generation unit (FGU), control room and/or examination room, patient table, patient model, coil, patient data display (PDD) and/or console with interactive GUI.

E.g., as the Microsoft Hololens (as XR headset or HMD) supports hand tracking with its build-in depth cameras, the user may freely interact with the (e.g., XR) scene, e.g., by clicking on the virtual GUI, select the patient to be examined, indication, and/or scan region. The clicking on the virtual GUI may also be denoted as input on a display (e.g., a patient data display, PDD) of the technical device (in particular the medical device) in XR, and/or as input on the GUI of a virtual operating console. Thereby, the real-world behavior of the GUI of a technical device, in particular a medical scanner and/or an operating console, may be mimicked.

The (e.g., XR) scene may implement the complete workflow. E.g., the user may grab objects (e.g., equipment of the MRI scanner, in particular a coil) and place them onto the patient model, position the patient and move the patient table into the isocenter of the MRI scanner by using functional buttons on the (in particular) MRI scanner.

The AR (and/or XR) scene is generated as realistically as possible, covering all necessary aspects so that the simulated and/or reconstructed training images at the end (e.g., see Step5below) are as realistic as possible. For the AR (and/or XR) scenes, the Photon Network technology (e.g., according to https://www.photonengine.com/pun) may be used, allowing multiple users to enter and/or interact with the same AR (and/or XR) Scene.

FIG.3shows the basic set-up. The operator306(e.g., wearing the Microsoft Hololens 2 as XR headset or HMD) is in a room302, and/or an arbitrary office space, and is recorded by three optical sensors304, in particular three RGB-D cameras (e.g., recording RGB and depth information of the scene). By having three optical sensors, e.g., three cameras, the operator can be covered from different angles in order to calculate a completely color-coded 3D point cloud of the operator306.

The inserted drawing in the lower right corner ofFIG.3shows the view through the operator's306Hololens. The XR (and/or AR) scene310inFIG.3comprises an MRI scanner312-1with a patient table312-2, on which a virtual patient314is situated. The MRI scanner312-1may be an exemplary embodiment of the technical device. The patient table312-2may be viewed as equipment of the MRI scanner312-1. Alternatively or in addition, the patient table312-2may be viewed as a further technical device.

The operator306is recorded by each of the three depth cameras304inFIG.3. The room302may comprise an arbitrary office space, e.g., having at least some empty floor area on which the operator306may move freely. Alternatively or in addition, the room302, as exemplified inFIG.3does not include any physical magnet, and/or components of the technical device (e.g., the MRI scanner), and/or equipment of the technical device.

Based on the depth images of the cameras304, a, e.g., color-coded, surface of the operator306may be reconstructed.

FIG.4shows an example of an image of a fused third dataset comprising the reconstructed operator306and the MRI scanner312-1as well as the patient table312-2, e.g., as comprised in the XR (and/or AR) scene310in the lower right corner ofFIG.3.

According to the inventive technique, the 3D depth data of the operator306are fused with the 3D surface data of the virtual scene310.FIG.4shows the final result, where one can see the operator306(as color-coded point cloud) fused with the virtual scene data as presented in the MS Hololens to the operator306.

FIG.4is an example of a synthetic image (in particular comprised in the third dataset generated by suing in the step S110of the method100), as this kind of image cannot be detected (and/or recorded) with any optical sensor (e.g., a camera, in particular a depth camera). The synthetic image ofFIG.4can only be obtained by fusing real and virtual world data, e.g., comprised in the second dataset and in the first dataset, respectively.

An example of a point cloud of the third dataset as a fusion of the first dataset and the second dataset displayed inFIG.5.

InFIG.5, the user (also denoted as operator)306of an MRI scanner312-2is shown in a first (e.g., checkered) style. The remaining (in particular XR) scene (e.g., comprising the MRI scanner312-1, patient table312-2with patient314and optical sensors304mounted on stands) is shown as points of different shade of gray, or different shade of color. For illustrative purposes, inFIG.5, light shades are denoted by empty circles, and dark shades are denoted by filled circles.

The inventive technique is powerful because it is capable of modeling, and/or displaying, any kind of user interaction of the operator306with the AR (and/or XR) scene, in particular with the technical device (e.g., with the MRI scanner312-1, and/or the patient table312-2).

Advantageously, with the inventive technique, all data are available as 5D datasets (e.g., comprising three spatial coordinates, a time-like coordinate, and a color; briefly: x, y, z, time, color). E.g., any one of the first dataset, the second dataset and the third dataset may comprise a 5D dataset. Any one of the datasets may comprise a fusion of partial datasets. E.g., the second dataset comprising optical sensor data may be obtained by fusing a plurality of optical sensor data, each of the optical sensor data corresponding to a different one of the optical sensors. Alternatively or in addition, the second dataset may comprise motion sensor data fused with the optical sensor data from the at least one optical sensor, in particular the at least one optical sensor at a fixed location relative to the room.

The (in particular full) 5D dataset cannot be shown in an (e.g., a single instantaneous) image. Alternatively or in addition, the 5D datasets may be viewed from any arbitrary angle of the scene (e.g., comprising the XR scene). Further alternatively or in addition, different kinds of visualizations may be calculated (e.g., synthetic depth images from any arbitrary angle).

Alternatively or in addition, an analysis (e.g., of the full 5D dataset) may be performed. The analysis can increase the amount of available test datasets for ML algorithms for scene understanding, e.g., by a multiplicative factor. The multiplicative factor may be at least 100, in particular at least 1000 or higher. Alternatively or in addition, different angles and/or different optical sensor (e.g., camera) positions may be comprised in the increased available test datasets.

In a second step (step2), a registration and/or calibration may be performed. By the registration and/or calibration, a correct (and/or accurate) overlay of the XR (and/or AR) scene with the real-world (in particular optical) sensor data may be enabled.

In order to achieve a high grade of registration quality, a dedicated registration and/or calibration routine may be implemented in the method100.

In order to register the (in particular optical) sensor data (also denoted as the real world) and the XR (and/or AR) scene (also denoted as the virtual world), a real, physical sphere602, which may be mounted on a (e.g., simple) camera stand and/or may be freely be moved around in the room, may be used as registration object, as exemplified for a plurality of registration objects inFIG.6A.

In a dedicated registration step (e.g., in an AR, and/or XR, application), virtual spheres604(as, in particular displayed, virtual registration objects) may be displayed at different positions (and/or locations) in the AR (and/or XR) scene (and/or in the virtual room). The virtual spheres604may be displayed to the user (and/or operator)306by a HMD or XR headset, e.g., via the MS Hololens2.

The user (and/or operator)306may be requested to move, e.g., a (e.g., a single one, or one out of the set) real, physical sphere602to one position of a (e.g., displayed) virtual sphere604in the AR (and/or XR) scene. The user (and/or operator)306may be instructed to move the real, physical sphere602to a predetermined position (and/or location) of a (e.g., displayed) virtual sphere604, e.g., according to a predetermined ordering of a set of (e.g., displayed) virtual spheres604.

According to one embodiment, a single virtual sphere604may be displayed at an instant in time of the registration step.

According to another embodiment, at least two (e.g., a set of) virtual spheres604may be displayed simultaneously during the registration step.

FIG.6Bshows how the user (and/or operator)306moved the real, physical spheres602to the positions of the displayed virtual spheres604. The real sphere602is seen overlayed by the virtual sphere604, e.g., in the HMD or XR headset, in particular in the MS Hololens.

InFIGS.6A and6B, more than one user (and/or operator) may be present. E.g., two different users (and/or operators) may be simultaneously present at locations denoted by reference sign606. Any one of the users (and/or operators) may be requested to move the real, physical spheres602. Alternatively or in addition, the registration procedure may comprise a synchronization of the views of the different users (and/or operators). The synchronization may, e.g., performed using a Photon Network (and/or PUN).

The real, physical spheres602may have one color, e.g., green. The displayed virtual spheres604may have another color, which differs from the color of the real, physical spheres604. E.g., the virtual spheres604may be displayed in the color orange.

The optical sensors may in particular comprise RGBD cameras.

From the RGB images of the available RGBD cameras (i.e., color image), a real, physical sphere602having the one color may be detected by computer vision methods, e.g., shape detection and color information. Alternatively or in addition, from the depth images of the same available RGBD cameras, the 3D coordinates of the real, physical sphere602with respect to the different cameras may be determined (e.g., calculated). E.g., a location (and/or position) of the center (e.g., the center of mass) of the real, physical sphere602may be determined.

Alternatively or in addition, a location (and/or position) of the outer shell of the real, physical sphere602may be determined.

The real, physical sphere602may have a diameter in the range of 50 mm to 200 mm, e.g., of 120 mm. The center (e.g., the center of mass) of the real, physical sphere602may be located at half the diameter into the interior of the sphere602from its outer shell. Alternatively or in addition, the outer shell of the real, physical sphere602may be located at half the diameter radially outwards from the center (e.g., the center of mass) of the real, physical sphere602.

The registration procedure may be repeated, e.g., according to a predetermined ordering, for all displayed virtual spheres604in the AR (and/or XR) scene.

A transformation matrix may be determined (e.g., calculated) to correct the transform of the optical sensors (e.g., cameras) in the AR (and/or XR) scene matching the real world data and virtual world data.

E.g., for each optical sensor, in particular camera, the position of the real, physical spheres in global coordinates of a computer vision system (e.g., Unity) scene may be stored in a matrix A and the position of a displayed virtual sphere may be stored in a matrix B:

Ai=[x^i,1y^i,1z^i,1………xi,4y^i,4z^i,4],i=1,2,B=[x1y1z1………x4y4z4].

Herein, i=1,2 denotes an exemplary numbering of optical sensors, in particular two cameras304. The second index 1,2,3,4 of the entries of the matrix Ai as well as the index 1,2,3,4 of the matrix B denotes the numbering of four physical spheres602.

The matrices Ai and B may be centered by subtracting a centroid. For simplicity of the presentation, the index i of Ai counting the optical sensor, in particular camera304, is suppressed in the following.
Acenter=A−CA, Bcenter=B−CB.

From the centered matrices Acenter and Bcenter, a covariance matrix may be computed:
MCov=Bcenter·AcenterT.

A singular value decomposition (SVD) may be applied to the covariance matrix:
U·S·VT=SVD(MCov),

with transformation matrices U and V and singular value decomposition (SVD) of the covariance matrix MCov.

From this, a rotation of the points of the matrix A may be determined:
R=U·VT.

Translations may be computed for each camera and its positions from:
t=CB−R·CA.

The centering of matrices, computing of the covariance matrix, computing of the SVD, as well as the resulting rotation R and translation t are performed each of a plurality of optical sensors, in particular cameras304, separately. Alternatively or in addition, the index i carries through the computation (e.g., Ai,center⇒Mi,Cov⇒Ui, Vi⇒Ri, ti).

A calibration of each of the optical sensors may comprise the registration of the registration objects, in particular, the registration spheres. Alternatively or in addition, a calibration of each of the optical sensors may comprise, based in the registration step, selecting (and/or adjusting) intrinsic camera parameters, and/or extrinsic camera parameters.

The visualization inFIG.4may use the registration and/or calibration according to the second step (step2). The user (and/or operator)306may, e.g., freely, move around in the room while observing (and/or) watching the AR (and/or XR) scene in the HMD or XR headset, in particular the MS Hololens. The user's (and/or operator's)306position in the synthetic (and/or fused) image (e.g., according to the third dataset) may be updated in real-time accordingly to the registration and/or calibration, and according to depth scene data (e.g., if the user, and/or operator,306stands in front, and/or behind the patient table and/or patient chair).

The advanced visualization according to the inventive technique, which takes the real-world data into account according to the second step (step2), allows for real-time displaying of the user (and/or operator306) interaction with the virtual (e.g., XR) scene for observers without a HMD or XR headset, e.g., not wearing the MS Hololens. Conventionally, only users (and/or observers) within the (e.g., XR) scene (e.g., using a HMD or XR headset, in particular wearing the MS Hololens) can see the interaction of the other users in the same (e.g., XR) scene.

The registration (and/or calibration) method of the second step (sept 2) may be extended to not only track and/or register real-world spheres (e.g., as used inFIGS.6A and6B), but may also track other real-world objects such as robotics and/or monitor arms, and/or (e.g., in-room) equipment of the technical device, in particular the medical device (e.g., the MRI scanner312-1) with computer vision methods. The computer vision methods may, e.g., comprise object detection, shape recognition, and/or ML based detectors.

With the extension of the registration (and/or calibration) method, e.g., overlaying new concepts for robotic devices on-top of the conventional holder structures may be enabled. Alternatively or in addition, new ways of interacting with robotic devices and/or monitors may be introduced, e.g., with the built-in gesture detection and/or hand tracking capabilities of the HMD or XR headset, e.g., extending the Microsoft Hololens 2. Alternatively or in addition, AR and/or XR may be extended, e.g., enabling early concept discussions and/or usability engineering for surgical procedures.

In a third step (step3), point-clouds are generated, e.g., virtual point-clouds for the first dataset, and/or point-clouds for the second dataset.

FIGS.7A to7Gshow an example of generating a virtual point-cloud as the third dataset.

InFIG.7Cas the first dataset, two virtual point-clouds fromFIGS.7A and7Bare fused, which correspond to different angles of view (e.g., corresponding to different angles of the optical sensors, in particular depth cameras).

InFIG.7Fas the second dataset, two point-clouds fromFIGS.7D and7Eare fused, which correspond to different angles of view of the optical sensor data (e.g., from the optical sensors, in particular RGBD cameras,304).

FIG.7Gshows the third dataset as a virtual point-cloud obtained from fusing of the virtual point-cloud of the first dataset ofFIG.7Cwith the point-cloud of the second dataset ofFIG.7F.

The inventive technique mainly builds upon using 3D point clouds as (e.g., XR) scene representation, e.g., per instant in time. The 3D point clouds may be generated from 3D surface data and/or computer-aided design (CAD) data of the AR (and/or XR scene (e.g., comprising a magnet and/or FGU of an MRI scanner, a patient table, and/or a patient model).

A patient model may be, or may comprise, a computer model comprising a surface mesh and applied textures (e.g., color images, overlayed onto the mesh data) to make the patient model look like a real patient (in particular a human). Similar models may be used in computer games or in 3D renderings for product presentations. A (e.g., comprehensive) database may be generated. Animations may be applied to the patient models to make the XR scene (also denoted as: act) more realistic, e.g., with the patient walking into the MRI room, and/or sitting on a patient table (and/or patient bed). Alternatively or in addition, real movement data of real patients may be recorded and/or applied the animation files of the patient models. E.g., a synthetically generated animation of a patient model may be modified based on the recorded real movement data.

To make the (e.g., XR) scene, and/or a patient movement, even more realistic, random movements may be added, e.g., to simulate a freezing patient and/or movement during patient preparation (e.g., when applying coils to a patient for an MRI scan).

The patient model may including animations (e.g., movements of a human patient) also be denoted as dynamic patient model.

By the (e.g., dynamic) patient models, a realism of the generated (e.g., XR) scenes can be improved. Alternatively or in addition, the (e.g., dynamic) patient models can positively influence (e.g., augment) the amount of available training data.

The generating of the point-clouds of the first dataset, and/or of the second dataset, can be done within a development framework for computer vision (e.g., Unity) by simple, e.g., scan-line algorithms to trace the surface of the objects within the scene. E.g.,FIG.7Cshows the generated point-cloud data of the AR (and/or XR) scene. Alternatively or in addition, e.g.,FIG.7Fshows the generated point-cloud data of the detected optical sensor data.

In a fourth step (step4), the (e.g., virtual) point-clouds of the first dataset are fused with the point-clouds of the optical sensor data, e.g., depth camera data, and/or of the second dataset.

The virtual (e.g., AR, and/or XR, scene) and real (operator, and/or optical sensor data, in particular depth camera) 3D point cloud data sets corresponding to the first dataset and the second dataset, respectively, are fused using the registration and/or calibration determined (e.g., calculated) in the second step (e.g., step2).FIG.7Gshows the result of the fusing and/or combining the point-clouds recorded from the depth cameras (e.g., as displayed inFIG.7F) and the ones as determined (e.g., calculated) from the virtual point-clouds of the XR (and/or AR) scene in the third step (step3), e.g., as displayed inFIG.7C.

It is noted that even if the references images are only shown in two-dimensions (2D) inFIG.5as well asFIGS.7A to7D, the fused point-cloud representation (e.g., as determined, and/or calculated, in the fourth step, step4) is a (e.g., true) 5D dataset (e.g., comprising spatial and time-like coordinates as well as color; briefly: x, y, z, time, color).

An example of the of fusing the point-clouds, including taking the color information for the operator306and/or the AR (and/or XR) scene (e.g., comprising an MRI scanner and a patient on a patient table) into account, is provided byFIG.5.

In a fifth step (step5), further determinations, calculations, and/or reconstructions may be performed on the fused third datasets, e.g., the point-clouds comprising a 5D representation with x,y,z,t coordinated and color information per data point, as, e.g., exemplified inFIG.5. E.g., if the user (and/or operator)306grabs an object, and/or equipment, in the scene, e.g., a coil (as an example of equipment of an MRI scanner) and moves the object, the 5D scene representation is updated accordingly.

The 5D scene representation, and/or the 5D data representation, allows for multiple ways for training data generation for ML, and/or for scene understanding, and/or analysis, algorithms.

Reconstructions may comprise a variety of different virtual, synthetic depth cameras from any arbitrary position, and/or any angle with different camera characteristics. E.g., synthetic depth camera images may be determined (and/or calculated) which a real RGBD camera would see in such a scene, e.g., if the inventive system is operationally installed (in particular, if at least one optical sensor and/or RGBD camera is installed in a room).

Alternatively or in addition, reconstructions may comprise a variety of RGBD images from any arbitrary position, angle, and/or size.

Further alternatively or in addition, reconstructions may comprise a variety of three-dimensional (3D) reconstructions.

The inventive technique in particular applies to a medical device, and more particularly to a medical imaging device (e.g., an MRI scanner312-1), as the at least one technical device in the XR scene, with which the user (and/or operator)306interacts. Alternatively or in addition, the at least one technical device may comprise robotics, and/or an autonomously driving vehicle, with which the user (and/or operator)306interacts according to the inventive technique.

The generated data (and/or, e.g., third, datasets) may be augmented, e.g., by animating the patient model, adding arbitrary movement, modifying the operator's306pose, and/or change position and/or angle for reconstructed images for ML. By augmenting the data (and/or, e.g., third, datasets), the amount of, e.g., third, datasets available for ML applications may be increased, e.g., by a multiplicative factor. The multiplicative factor may be at least 1000, in particular at least 10000 or higher.

An ML algorithm may be applied to the, e.g., third, datasets in order to generate training and/or test data. Alternatively or in addition a ML based algorithm may be developed for scene understanding and/or analysis.

The ML algorithm, and/or the reconstructions, may comprise a semantic scene segmentation and/or understanding. Alternatively or in addition, the ML algorithm, and/or the reconstructions, may comprise a human pose detection. E.g., the pose of the user (and/or operator)306may be detected. Alternatively or in addition, a patient's skeleton, and/or joints may be detected. Further alternatively or in addition, the ML algorithm, and/or the reconstructions, may comprise a pose tracking and/or action prediction, e.g., learning the action the user (and/or operator)306is about to perform in the (e.g., XR) scene. Still further alternatively or in addition, the ML algorithm, and/or the reconstructions, may comprise a workflow step detection and/or classification, e.g., for clinical supervision.

The ML based algorithm may be trained using the presented AR (and/or XR) framework and data representation (e.g., comprising the first dataset, second dataset, and/or third dataset).

By the inventive technique, an autonomy of a technical device, in particular a medical device (and more particularly a medical imaging device, e.g., an MRI scanner), and/or its awareness of, and/or interaction with, the environment may be improved. The environment may, e.g., comprise the user (and/or operator)306. Alternatively or in addition, the environment may comprise a patient to be examined.

FIG.8shows two exemplary alternative uses of applying a CycleGAN for image-to-image transfer and subsequently applying a (in particular different from the CycleGAN) neural network, in particular an RfD-Net, for determining a semantic context, when fusing a first dataset806representing an XR scene with at least one technical device, in particular a medical device, and a second dataset808comprising (in particular optical) sensor data indicative of a user's interaction with the technical device, in particular the medical device.

According to the first alternative802, the first dataset806and the second dataset808are fused at reference sign812in order to obtain a third dataset810. The third dataset810undergoes image-to-image transfer via the CycleGAN as shown at reference sign814. The output of the CycleGAN is a modified third dataset810-1, which is, as shown at reference sign816, input into a (in particular different from the CycleGAN) neural network, e.g., a RfD-Net. At reference sign810-2, a further modified third dataset is output comprising a semantic context.

According to the second alternative804, the first dataset806is input, as shown at reference sign814, into the CycleGAN, to obtain an image-to-image transferred (also denoted as modified) first dataset806-1. The modified first dataset806-1and the second dataset808are fused at reference sign812to obtain a third dataset810, which is input, as shown at reference sign816, into a (in particular different from the CycleGAN) neural network, e.g., a RfD-Net. The output of the neural network, e.g., the RfD-Net, at reference sign810-2comprises a modified third dataset with a semantic context.

FIG.9shows a detailed example of generating a more realistic (e.g., photorealistic) image from a virtual image, e.g., corresponding to the first dataset, using a CycleGAN.

FIG.9Ashows a depth image of an MRI scanner312-1with a patient table312-2as an example of a technical device (and/or a combination of two technical devices) as obtained from a real depth camera, e.g., an RGBD camera.

FIG.9Bshows an exemplary generated synthetic depth image, e.g., using a shader in a computer vision system (e.g., Unity), of a scene with similar geometries as inFIG.9A. At reference sign902, depth images obtained from a real depth camera, and/or generated synthetic depth images, are used to train a CycleGAN.

FIG.9Cexemplifies the step814of applying a CycleGAN for image-to-image transfer to the synthetic depth image ofFIG.9Bin order to arrive at a modified (e.g., more realistic and/or photorealistic) version of the image, comprising the MRI scanner312-1and the patient table312-2, inFIG.9D.

FIG.10shows an exemplary embodiment of the workings of a system comprising a computing device200, a multitude of HMDs and/or XR headsets (e.g., Hololenses)204-1;204-2;204-3and optical sensors304.

As depicted as reference sign1002, the computing device200may comprise a computer vision system (e.g., Unity)1002, which through a TCP Client/Service1012connects to a software and/or programming language (e.g., Python) interface1004, which over the second input interfaces202-2connects to the optical sensors304.

The multitude of HMDs and/or XR headsets (e.g., Hololenses)204-1;204-2;204-3inFIG.10are connected among each other as well as with the computing device200through the PUN1010.

The output of the computer vision system (e.g., Unity)1002comprises one or more virtual point clouds1002.

The output of the software and/or programming language (e.g., Python) interface1008comprises one or more real point clouds1008.

The (e.g., complete) 3D scene (e.g., comprising the third dataset) may be implemented in a computer vision system (e.g., Unity) based on computer-aided design (CAD) models.

By the inventive technique, a development (also denoted as product development) of technical devices, in particular of medical devices (and more particularly medical imaging devices, e.g., of MRI scanners) may be improved, rendered more efficient, and/or accelerated.

Alternatively or in addition, by the inventive technique, synthetic data generation may be improved (e.g., using the third dataset), and/or ML based algorithms and/or automatisms may be improved, and/or developed.

The inventive technique far exceeds the conventional Microsoft Mesh and Holoportation projects as well as the conventional developments in the automotive drive and/or robotics industry (e.g., NVIDIA's simulation frameworks for synthetic data generation for ML based applications) by incorporating the operator's (and/or user's) interactions. Alternatively or in addition, workflow modeling, in particular comprising a medical workflow, may be improved, e.g., in terms of a patient's safety (e.g., by choosing the least invasive examination method, a minimal amount of radiation, and/or a minimum magnetic field strength, and/or by ensuring that an implant, e.g., a cardiac pacemaker, is correctly taken into account) and/or speed of an examination.

The Inventive technique allows for very early customer involvement and/or feedback with a high level of immersion (e.g., via rendering an image of the third dataset, and/or by a user performing virtual tests on the operation of the at least one technical device in the XR scene) during the development of a technical device. Alternatively or in addition, training data for scene understanding and/or analysis for the technical device may already be generated during the development, construction and/or assembly phase of the real, physical technical device. Further alternatively or in addition, the development of novel technical devices may be de-risked. E.g., by virtually monitoring, and/or controlling the operating of a dental MRI scanner under development, it may be ensured that all (e.g., confined) space and operational (e.g., reachability) constraints may be met.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, components, regions, layers, and/or sections, these elements, components, regions, layers, and/or sections, should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and/or,” includes any and all combinations of one or more of the associated listed items. The phrase “at least one of” has the same meaning as “and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below,” “beneath,” or “under,” other elements or features would then be oriented “above” the other elements or features. Thus, the example terms “below” and “under” may encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. In addition, when an element is referred to as being “between” two elements, the element may be the only element between the two elements, or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example, between modules) are described using various terms, including “on,” “connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the disclosure, that relationship encompasses a direct relationship where no other intervening elements are present between the first and second elements, and also an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. In contrast, when an element is referred to as being “directly” on, connected, engaged, interfaced, or coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between,” versus “directly between,” “adjacent,” versus “directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a,” “an,” and “the,” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the terms “and/or” and “at least one of” include any and all combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list. Also, the term “example” is intended to refer to an example or illustration.

It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It will be further understood that terms, e.g., those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

It is noted that some example embodiments may be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented in conjunction with units and/or devices discussed above. Although discussed in a particularly manner, a function or operation specified in a specific block may be performed differently from the flow specified in a flowchart, flow diagram, etc. For example, functions or operations illustrated as being performed serially in two consecutive blocks may actually be performed simultaneously, or in some cases be performed in reverse order. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed, but may also have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. The present invention may, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

In addition, or alternative, to that discussed above, units and/or devices according to one or more example embodiments may be implemented using hardware, software, and/or a combination thereof. For example, hardware devices may be implemented using processing circuity such as, but not limited to, a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a System-on-Chip (SoC), a programmable logic unit, a microprocessor, or any other device capable of responding to and executing instructions in a defined manner. Portions of the example embodiments and corresponding detailed description may be presented in terms of software, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, or as is apparent from the discussion, terms such as “processing” or “computing” or “calculating” or “determining” of “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device/hardware, that manipulates and transforms data represented as physical, electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

In this application, including the definitions below, the term ‘module’ or the term ‘controller’ may be replaced with the term ‘circuit.’ The term ‘module’ may refer to, be part of, or include processor hardware (shared, dedicated, or group) that executes code and memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware.

The module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing. In a further example, a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module.

Software may include a computer program, program code, instructions, or some combination thereof, for independently or collectively instructing or configuring a hardware device to operate as desired. The computer program and/or program code may include program or computer-readable instructions, software components, software modules, data files, data structures, and/or the like, capable of being implemented by one or more hardware devices, such as one or more of the hardware devices mentioned above. Examples of program code include both machine code produced by a compiler and higher level program code that is executed using an interpreter.

For example, when a hardware device is a computer processing device (e.g., a processor, Central Processing Unit (CPU), a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a microprocessor, etc.), the computer processing device may be configured to carry out program code by performing arithmetical, logical, and input/output operations, according to the program code. Once the program code is loaded into a computer processing device, the computer processing device may be programmed to perform the program code, thereby transforming the computer processing device into a special purpose computer processing device. In a more specific example, when the program code is loaded into a processor, the processor becomes programmed to perform the program code and operations corresponding thereto, thereby transforming the processor into a special purpose processor.

Software and/or data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, or computer storage medium or device, capable of providing instructions or data to, or being interpreted by, a hardware device. The software also may be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion. In particular, for example, software and data may be stored by one or more computer readable recording mediums, including the tangible or non-transitory computer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the form of a program or software. The program or software may be stored on a non-transitory computer readable medium and is adapted to perform any one of the aforementioned methods when run on a computer device (a device including a processor). Thus, the non-transitory, tangible computer readable medium, is adapted to store information and is adapted to interact with a data processing facility or computer device to execute the program of any of the above mentioned embodiments and/or to perform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented in conjunction with units and/or devices discussed in more detail below. Although discussed in a particularly manner, a function or operation specified in a specific block may be performed differently from the flow specified in a flowchart, flow diagram, etc. For example, functions or operations illustrated as being performed serially in two consecutive blocks may actually be performed simultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processing devices may be described as including various functional units that perform various operations and/or functions to increase the clarity of the description. However, computer processing devices are not intended to be limited to these functional units. For example, in one or more example embodiments, the various operations and/or functions of the functional units may be performed by other ones of the functional units. Further, the computer processing devices may perform the operations and/or functions of the various functional units without sub-dividing the operations and/or functions of the computer processing units into these various functional units.

Units and/or devices according to one or more example embodiments may also include one or more storage devices. The one or more storage devices may be tangible or non-transitory computer-readable storage media, such as random access memory (RAM), read only memory (ROM), a permanent mass storage device (such as a disk drive), solid state (e.g., NAND flash) device, and/or any other like data storage mechanism capable of storing and recording data. The one or more storage devices may be configured to store computer programs, program code, instructions, or some combination thereof, for one or more operating systems and/or for implementing the example embodiments described herein. The computer programs, program code, instructions, or some combination thereof, may also be loaded from a separate computer readable storage medium into the one or more storage devices and/or one or more computer processing devices using a drive mechanism. Such separate computer readable storage medium may include a Universal Serial Bus (USB) flash drive, a memory stick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other like computer readable storage media. The computer programs, program code, instructions, or some combination thereof, may be loaded into the one or more storage devices and/or the one or more computer processing devices from a remote data storage device via a network interface, rather than via a local computer readable storage medium. Additionally, the computer programs, program code, instructions, or some combination thereof, may be loaded into the one or more storage devices and/or the one or more processors from a remote computing system that is configured to transfer and/or distribute the computer programs, program code, instructions, or some combination thereof, over a network. The remote computing system may transfer and/or distribute the computer programs, program code, instructions, or some combination thereof, via a wired interface, an air interface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices, and/or the computer programs, program code, instructions, or some combination thereof, may be specially designed and constructed for the purposes of the example embodiments, or they may be known devices that are altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run an operating system (OS) and one or more software applications that run on the OS. The computer processing device also may access, store, manipulate, process, and create data in response to execution of the software. For simplicity, one or more example embodiments may be exemplified as a computer processing device or processor; however, one skilled in the art will appreciate that a hardware device may include multiple processing elements or processors and multiple types of processing elements or processors. For example, a hardware device may include multiple processors or a processor and a controller. In addition, other processing configurations are possible, such as parallel processors.

The computer programs include processor-executable instructions that are stored on at least one non-transitory computer-readable medium (memory). The computer programs may also include or rely on stored data. The computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc. As such, the one or more processors may be configured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language) or XML (extensible markup language), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C #, Objective-C, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5, Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one example embodiment relates to the non-transitory computer-readable storage medium including electronically readable control information (processor executable instructions) stored thereon, configured in such that when the storage medium is used in a controller of a device, at least one embodiment of the method may be carried out.

The computer readable medium or storage medium may be a built-in medium installed inside a computer device main body or a removable medium arranged so that it can be separated from the computer device main body. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium is therefore considered tangible and non-transitory. Non-limiting examples of the non-transitory computer-readable medium include, but are not limited to, rewriteable non-volatile memory devices (including, for example flash memory devices, erasable programmable read-only memory devices, or a mask read-only memory devices); volatile memory devices (including, for example static random access memory devices or a dynamic random access memory devices); magnetic storage media (including, for example an analog or digital magnetic tape or a hard disk drive); and optical storage media (including, for example a CD, a DVD, or a Blu-ray Disc). Examples of the media with a built-in rewriteable non-volatile memory, include but are not limited to memory cards; and media with a built-in ROM, including but not limited to ROM cassettes; etc. Furthermore, various information regarding stored images, for example, property information, may be stored in any other form, or it may be provided in other ways.

The term code, as used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. Shared processor hardware encompasses a single microprocessor that executes some or all code from multiple modules. Group processor hardware encompasses a microprocessor that, in combination with additional microprocessors, executes some or all code from one or more modules. References to multiple microprocessors encompass multiple microprocessors on discrete dies, multiple microprocessors on a single die, multiple cores of a single microprocessor, multiple threads of a single microprocessor, or a combination of the above.

Shared memory hardware encompasses a single memory device that stores some or all code from multiple modules. Group memory hardware encompasses a memory device that, in combination with other memory devices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium is therefore considered tangible and non-transitory. Non-limiting examples of the non-transitory computer-readable medium include, but are not limited to, rewriteable non-volatile memory devices (including, for example flash memory devices, erasable programmable read-only memory devices, or a mask read-only memory devices); volatile memory devices (including, for example static random access memory devices or a dynamic random access memory devices); magnetic storage media (including, for example an analog or digital magnetic tape or a hard disk drive); and optical storage media (including, for example a CD, a DVD, or a Blu-ray Disc). Examples of the media with a built-in rewriteable non-volatile memory, include but are not limited to memory cards; and media with a built-in ROM, including but not limited to ROM cassettes; etc. Furthermore, various information regarding stored images, for example, property information, may be stored in any other form, or it may be provided in other ways.

The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks and flowchart elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

Although described with reference to specific examples and drawings, modifications, additions and substitutions of example embodiments may be variously made according to the description by those of ordinary skill in the art. For example, the described techniques may be performed in an order different with that of the methods described, and/or components such as the described system, architecture, devices, circuit, and the like, may be connected or combined to be different from the above-described methods, or results may be appropriately achieved by other components or equivalents. Wherever not already described explicitly, individual embodiments, or their individual aspects and features, described in relation to the drawings can be combined or exchanged with one another without limiting or widening the scope of the described invention, whenever such a combination or exchange is meaningful and in the sense of this invention. Advantages which are described with respect to a particular embodiment of present invention or with respect to a particular figure are, wherever applicable, also advantages of other embodiments of the present invention.