Patent ID: 12206837

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

An embodiment of the present invention provides a near eye assembly having a retaining structure that is configured to be positioned in proximity to the eye of a user of the assembly. Typically, the retaining structure comprises a spectacle frame. Alternatively, the retaining structure comprises a head up-display which may be mounted on a helmet worn by the assembly user.

An optical combiner is mounted on the structure in front of the user eye. Typically, two combiners are mounted, one in front of each eye. The optical combiner at least partially transmits elements of a scene in front of the assembly through the combiner. In addition, the optical combiner may receive a visible radiation transmission derived from a scene, and/or a visual transmission such as a presentation of data or a marker, and redirects the transmission back to the user's eye.

A pixelated screen, comprising an array of variably transparent pixels, coats the optical combiner. Typically, the pixels are liquid crystal display (LCD) pixels.

There is at least one image capturing device, typically two such devices, one for each eye, mounted on the structure. The capturing device is typically a visible spectrum camera that is configured to capture an image of a scene viewed by the user's eye.

A projector, typically a micro-projector, is mounted on the structure. Typically two projectors, one for each eye, are mounted on the structure. The projector is configured to project at least one of a portion of the captured image as a video, as well as a stored image, onto a section of the screen that a processor renders at least partially opaque. The at least partially opaque section is also referred to herein as an occlusion mask, or just as a mask.

The processor is configured to select the location of the section in response to a region of interest in the scene identified by analysis of the captured image. Typically, at least one marker is positioned near the region of interest, and the processor analyzes the captured image to locate the marker and so identify the region of interest. Rendering the section opaque occludes the region of interest from the user's eye.

In addition, the processor determines a dimension of the section, typically, in the case of the section being circular, the diameter of the section. The dimension is determined in response to the pupil diameter.

By setting the dimension of the section according to the pupil diameter, embodiments of the present invention more exactly control the area of the region of interest that is occluded. In addition, because of the finite size of the pupil, there is a region surrounding region of interest that is partially occluded. In some embodiments the processor operates the micro-projector to overlay relevant portions of the captured image on the partially occluded region, so as to compensate for the partial occlusion.

As stated above, a portion of the captured image may be projected as a video onto the occlusion mask. In some embodiments the captured image portion video corresponds to the occluded region of interest. There is a non-occluded region surrounding the occluded region of interest, and this non-occluded region is visible to the user through the combiner. In embodiments of the invention the video and the visible non-occluded region are typically not in accurate registration, due to slight inevitable movements of the display relative to the user's eye.

In some embodiments a stored image, such as an image of a tool, is overlaid on, and in accurate registration with, the occluded region video.

The inventors have found that registering the stored image with the video, even though the video is not fully registered with the surrounding visible region, provides an acceptable image for the user. The inventors have found that for a non-occluded region that appears to be 50 cm from the user's eye, the video and the non-occluded region may be out of registration by up to 2 cm, while still being acceptable to the user.

Thus, in contrast to prior art augmented reality systems, embodiments of the present invention are configured to operate with mis-alignment between the visible portion of a scene and an augmented reality portion of the scene. However, there is no mis-alignment between elements within the augmented reality video, i.e., the elements projected onto the occlusion mask.

In some embodiments, the optical combiner may be rotated about an axis by the processor. In the case of two combiners, they may be independently rotated about respective axes. The independent rotations may be used to orient both combiners so that each is orthogonal to the direction of gaze of the user's eyes.

System Description

Reference is now made toFIG.1, which schematically illustrates use of an augmented reality system20, according to an embodiment of the present invention. By way of example and for simplicity, in the following description system20is assumed to be used in a medical procedure during part of which the user of the system is being mentored. However, it will be understood that embodiments of the present invention may be used in non-medical and/or non-mentoring situations, such as in operating a video game, in simulating a real-world event, or in providing an aid to navigation.

System20is operated by a medical professional22, who wears an augmented reality assembly24, described in more detail below with respect toFIGS.2A-2D. While assembly24may be incorporated for wearing into a number of different retaining structures on professional22, in the present description the retaining structure is assumed to be similar to a pair of spectacles. Those having ordinary skill in the augmented reality art will be aware of other possible structures, such as incorporation of the augmented reality assembly into a head-up display that is integrated into a helmet worn by the user of system20, and all such structures are assumed to be comprised within the scope of the present invention.

System20comprises and is under overall control of a processor26. In one embodiment processor26is assumed to be incorporated within a stand-alone computer28, and the processor typically communicates with other elements of the system, including assembly24, wirelessly, as is illustrated inFIG.1. Alternatively or additionally, processor26may use optical and/or conducting cables for communication. In further alternative embodiments processor26is integrated within assembly24, or in the mounting of the assembly. Processor26is typically able to access a database40, wherein are stored images and other visual elements used by system20. Software enabling processor26to operate system20may be downloaded to the processor in electronic form, over a network, for example. Alternatively or additionally, the software may be provided on non-transitory tangible media, such as optical, magnetic, or electronic storage media.

The medical procedure exemplified here is on a patient30, and during the procedure professional22gazes along gaze directions32at a region of interest (ROI)34. ROI34typically, but not necessarily, comprises a portion of the patient. In some embodiments one or more ROI acquisition markers35, comprising marker elements36, are positioned in, and/or in proximity to, ROI34, and the functions of such markers are described below. Typically there are at least three marker elements36for a given marker35. In a disclosed embodiment the size of ROI34may be predefined by professional22, for example based on a computerized tomography (CT) image of the patient, and the position of the ROI may also be a predefined distance to the right and a predefined distance below the marker. In an alternative embodiment marker elements36of marker define ROI34to be a region within a surface having elements36in the perimeter of the surface. Typically, a margin in an approximate range of 1-5 cm is added to ROI34to compensate for mis-alignment between a video projection and a directly viewed scene, described in more detail below.

During the procedure professional22may use a surgical device38, such as a surgical knife, to perform part of the procedure. Typically device38comprises one or more identifying elements39which may be used to track the device.

FIGS.2A-2Eare schematic diagrams illustrating assembly24, as well as functions that may be implemented in the assembly, according to an embodiment of the present invention.FIG.2Aillustrates assembly24with none of the active elements of the assembly, i.e., those elements requiring power, operating. As stated above, assembly24is configured, by way of example, as a pair of spectacles50. Similar elements of each “half” of the pair of spectacles are referred to generically by an identifying numeral, and the similar elements are differentiated as necessary by adding a letter to the numeral. Thus spectacles50comprise planar optical combiners52, comprising combiners52A and52B in front of, respectively, the left and right eyes of professional22. Optical combiners52are mounted on a retaining structure54which holds elements of assembly24, and which is herein assumed to comprise a spectacle frame, so that structure54is also referred to herein as frame54.

In some embodiments, combiner frames82A and82B are fixed to retaining structure54and vertical retaining rods84A and84B attached to the combiner frames support the optical combiners, so that the combiners are able to rotate about vertical axes defined by the rods. Retaining rods84A and84B, and thus combiners52A and52B, may be rotated independently of each other about their vertical axes by respective motors86A and86B, fixed to frames82A and82B. Motors86, typically stepper motors, are controlled by processor26so as to rotate their attached combiners to known, typically different, fixed orientations with respect to their respective combiner frames.

Each optical combiner52is configured to at least partially transmit elements of a scene through the combiner, so that a portion56of patient30(FIG.1) is assumed to be directly visible through each combiner52. In addition, each optical combiner52is configured to receive a visible radiation transmission derived from a scene, and/or a visual transmission such as a presentation of data or a marker, and to redirect or reflect the transmission back to the eye of professional22. The redirection is such that the scene and/or data or marker presented to the professional appears to be at a distance between the near and far points of vision of the professional. Thus, any given section of the optical combiner may combine directly visible material with redirected or reflected material, and provide this combined material to the eye of the user. More detail of the functioning of combiners52is provided below.

Optical combiners of various types are known in the art. One known type uses a semi reflective surface which transmits an image from an image source after it has passed through a set of lenses which correct deformations caused by the semi reflective surface of the combiner. Another known type uses a waveguide which projects the image directly to the eye of the viewer. Herein, by way of example, combiners52are assumed to be of the waveguide type.

In one embodiment, combiners52comprise LUMUS DK 32 see through glasses, produced by Lumus Optical of Rechovot, Israel.

Generally similar pixelated variable transparency screens60A and60B respectively coat a rear side, i.e., the side away from the eyes of professional22, of combiners52A,52B. Screens60are active elements of system20and are formed of an array of pixels, the opacity of each of the pixels being controlled by processor26.

Screens60are typically, but not necessarily, liquid crystal displays (LCDs) formed of a rectangular array of liquid crystal pixels. Alternatively, screens60are formed of MEMS (microelectromechanical systems). Further alternatively, screens60are formed of polymer dispersed liquid crystals (PDLCs). In the following description, by way of example, screens60are assumed to be formed of LCDs. LCD display pixels can typically be switched between an opaque state, where approximately 95% of the incoming light is blocked and 5% is transmitted, and a transparent state where approximately 60% of the incoming light is blocked and 40% is transmitted. The LCDs then have a transmission contrast ratio of 1:8.

Fixedly attached to arms of frame54are generally similar micro-projectors64A and64B. Each micro-projector is located and oriented so as to be able to project onto respective combiner52A and52B, a scene, and/or a visual indication, in a form suitable for redirection by the combiners to the left or right eye of professional22. Micro-projectors64are active elements, and the projected scenes/indications are provided to the micro-projectors by processor26. The projection and redirection are configured so that the images seen by the eyes of professional22, absent any correcting lenses, appear to be at infinity, due to parallel light coming from the combiners and entering the pupils. In some embodiments display24comprises correcting lenses88A,88B which redirect light from combiners52A,52B so that the images appear to be closer than infinity to the professional's eyes. The power D in diopters of the lenses defines the distance d of the images, according to the formula d=1/D, where d is in meters, and D is a negative number. Lenses88A,88B are typically located between the professional's eyes and the respective combiners. For simplicity, lenses88A,88B are not shown in other figures of the present application.

At least one image capturing device68is attached to frame54. In the disclosed embodiment there are two generally similar devices68A and68B, respectively aligned to be approximately orthogonal to planar combiners52A and52B, so as to be able to capture radiation of respective images of scenes viewed by the left and right eyes of professional22. Typically, devices68comprise cameras configured to capture images of scenes in the visible spectrum. The cameras may use rolling shutters, in which cases latency (of projection via micro-projectors64) may be reduced by processing rows of images rather than complete frames of images. In some embodiments devices68may also capture non-visible portions of images, such as portions in the infra-red spectrum. The operation of devices68is controlled by processor26.

In some embodiments of the present invention, assembly24comprises a sensor72which is configured to capture non-visible images of elements of a scene in front of assembly24. Typically sensor72uses a projector73configured to project radiation in the non-visible spectrum detected by the sensor, and has a bandpass filter configured to block visible radiation, such as that projected by surgical lighting. Typically, sensor72and projector73operate in the near infra-red spectrum.

In some embodiments, assembly24comprises a manual and/or electronic control74which may be operated by professional22to move elements of the assembly in and out of the field of view of the professional. Additionally or alternatively, there may be a button or switch78which enables the professional to power active elements of assembly24, such as the capturing devices and the micro-projectors. In some embodiments switch78may be a foot switch. Further additionally or alternatively, assembly24may be configured so that it can tilt downwards about a horizontal axis, at an angle up to 400 from the horizontal, so that the professional can look through the assembly when looking down.

Additionally, assembly24may comprise a sensor76, such as an accelerometer, which is configured to measure an inclination of the assembly with respect to the direction of gravity, so measuring the angle of the head of the professional with respect to the vertical. Processor26may be configured to use readings from sensor76to move elements of assembly24in and out of the field of view of the professional, and/or to control whether micro-projectors64project images.

FIG.2Bschematically illustrates the appearance of assembly24when processor26activates screens60A and60B. As described above, each screen60comprises an array of pixels, and the opacity of each pixel in an individual screen may be set by processor26. In screen60A the processor has rendered a circular array80A of the pixels of the screen opaque, while the remaining pixels of the screen are rendered transparent. The opacity of array80A means that from the point of view of the left eye of professional22, circular array80A acts as a mask occluding corresponding features of portion56of the patient, so that array80A is also referred to herein as occluding mask80A.

Similarly in screen60B processor26has rendered a circular array80B of the pixels of the screen opaque, while the remaining pixels of the screen are rendered transparent. As for array80A, array80B occludes sections of portion56from the view of the right eye of professional22. Thus array80B is also referred to herein as occluding mask80B.

FIG.2Cschematically illustrates the appearance of assembly24when processor26activates screens60and micro-projectors64A and64B. Screens60are activated to provide occluding masks80A and80B, as described above with respect toFIG.2B. Micro-projector64A projects a prerecorded ultrasound image90A of the patient's abdomen so as to overlay the image on mask80A, and micro-projector64B projects an image90B of the abdomen so as to overlay it on mask80B. Typically, although not necessarily, images90A and90B are the same. In some cases, for example if the images have been acquired in a stereoscopic manner or for correct 3D perception, images90A and90B may be slightly different, typically being slightly displaced horizontally with respect to each other. Micro-projectors64are configured to position images90A and90B on their respective masks so that, as seen by professional22and with −2 diopter lenses88A,88B present, the images are in focus at approximately 50 cm and appear to be at the location of the patient's abdomen.

In addition to projecting images90, micro-projectors64also project alphanumeric data92A and92B onto the non-occluded region of screens60, as well as markers96A and96B onto masks80A and80B. Images90, data92, and markers96are typically stored in database40, and are provided from the database to micro-projectors64by processor26.

In a mentoring situation images90, the contents of data92, and the position of markers96are typically under control of a tutor interacting with processor26while mentoring professional22. In some cases the locations of masks80may also be provided to processor26by the tutor, although typically the locations of the masks depend upon gaze directions32of the professional. In a non-mentoring situation, i.e. where professional22alone operates system20, locations of masks80are typically automatically set by processor26, as is described below. Also in a non-mentoring situation, images90, data92, and markers96may be controlled by professional22. It will be understood that images90, data92and markers96are examples of non-video related visual elements that are seen by professional22, and that the provision of such elements corresponds to an optic-based augmented reality situation implemented in system20.

FIG.2Dschematically illustrates the appearance of assembly24when processor26activates screens60and micro-projectors64, and in addition incorporates a video-based augmented reality feature into the operation of the assembly. Screens60and micro-projectors64are assumed to be activated as described above forFIG.2C, so that masks80, images90, data92and markers96are in the field of view of professional22. By way of example, the figure has been drawn to illustrate a mentoring situation, where the tutor of professional22wants to point to a feature of the chest of patient30, herein assumed to comprise an unusual movement of the chest.

To point to the feature, the tutor interacts with processor26so that the processor enhances and emphasizes portions100A,100B of the video images acquired by capturing devices68, the portions corresponding to the region of the chest where the unusual movement is occurring. Micro-projectors64A,64B then project portions100A,100B onto combiners52A,52B. It will be understood that the enhancement of portions100A,100B and their projection on the respective combiners is in real-time. The enhancement may take a number of forms. For example, portions100A,100B may comprise a wireframe image of the region of the chest having unusual movement, and/or a false-color image of the region. Other suitable methods of real-time enhancement will be apparent to those having ordinary skill in the art, and all such methods are assumed to be within the scope of the present invention.

FIG.2Eschematically illustrates an overall scene101as seen by professional22, during an invasive surgical procedure being performed by the professional. For simplicity,FIG.2Eillustrates the scene as it is presented on combiner52A, and it will be understood that a substantially similar scene is presented to the professional on combiner52B. The figure illustrates a hand102of professional22holding device38, herein assumed to comprise a pair of tweezers, at a proximal end of the device. One or more device identifying elements39, typically reflectors and/or radiators, are attached to the tweezers, so that processor26is able to identify and track device38using images acquired by capturing devices68.

The professional has made an incision104in a portion106of patient30, and ROI34, defined by marker elements36, is assumed to be at the location of the incision. In addition, the professional has inserted a lower portion of the distal end of device38into the patient so that the lower portion is no longer visible.

Processor26has formed mask80A on combiner52A so as to occlude ROI34, and the portion of incision104comprised in the ROI. Mask80A also includes a margin83, typically corresponding to a margin of approximately 1-5 cm at the ROI. Thus, all elements of the scene outside mask80A, comprising hand102and the proximal end of device38, are directly visible through combiner52A by the professional. However, elements of the scene within mask80A, including a portion of incision104and an upper portion of the distal end of device38that is outside the patient, are not visible to the professional, since they are occluded by the mask.

Processor26overlays on mask80A a captured image110of the ROI and the region corresponding to margin83, which includes the portion of incision104occluded by the mask and which also includes a video image114of the upper portion of the distal end of device38(outside the patient) that has been captured by image capturing device68. In addition, the processor overlays on the occlusion mask a stored image112corresponding to the lower portion of the distal end of device38(within the patient). Stored image112is a virtual elongation of image114and is retrieved from database40. The section of the distal end corresponding to image112is not visible to capturing device68.

The processor registers the two overlaid images, image110and image112, with each other, and the registration is possible since by tracking device38the processor is aware of the location of the device distal end with respect to the captured image. Thus, there is no misalignment between stored image112, corresponding to the lower portion of the distal end, and image114of the upper portion of the distal end, which is included in captured image110.

However, there is typically misalignment between the two registered overlaid images110,112and the directly visible portion of scene101, including the directly visible portion of incision104, as is illustrated in the figure. The misalignment occurs because while the captured image of the ROI is close to that seen by the professional (in the absence of the occlusion mask), it is not exactly in registration with the viewed scene. The inventors have found that a misalignment of up to 2 cm, in a scene that is 50 cm from the eye of the professional, is acceptable.

FIGS.3A and3Bare schematic diagrams illustrating assembly24in different orientations with respect to ROI34,FIG.3Cis a schematic diagram illustrating angles of the assembly for the different orientations, andFIGS.3D,3Eare graphs of the angles, according to an embodiment of the present invention. For simplicity, combiner frames82are not shown in the diagrams. InFIGS.3A and3Bprocessor26has positioned masks80so that they act to occlude ROI34from eyes120A,120B of professional22, specifically from pupils124A,124B of the professional's eyes.FIG.3Aillustrates a situation where ROI34is approximately directly in front of professional22. In this case the processor forms mask80A′ to be on a straight line with pupil124A and ROI34, while simultaneously forming mask80B′ to be on a straight line with pupil124B and the region of interest.

FIG.3Billustrates a situation where ROI34is not directly in front of professional22, but is towards one side of the professional. As for the situation ofFIG.3Athe processor forms mask80A″ to be on a straight line with pupil124A and ROI34, and forms mask80B″ to be on a straight line with pupil124B and the region of interest. In all cases masks80act as occlusion masks, and, as is illustrated by the differences in the positions of the masks, the processor changes the locations of the masks to account for changes in orientation of the region of interest with respect to assembly24.

A ring130surrounding ROI34is described in more detail below.

FIG.3Cschematically illustrates the two situations ofFIGS.3A and3B, when ROI34is at a distance L from eyes120A,120B of the professional. Eyes120A and120B are separated by a distance w. For the situation ofFIG.3A, where the region of interest is directly in front of the professional, ROI34is at a location103. For the situation ofFIG.3B, ROI34is to the left of the professional, at a location104that is a distance R from location103.

For the first situation, where professional22is looking at location103, the directions of gaze, αR, αLof the professional are shown by lines103R and103L. αR, αLare angles that are measured with respect to lines orthogonal to a line connecting eyes120A,120B, and their values are given by the following equations:

αL=-arc⁢tan⁡(w2⁢L),αR=+arc⁢tan⁡(w2⁢L)(A)

For the first situation processor26rotates combiners52A and52B (for clarity the combiners are not shown in the figure for the first situation), within their respective frames80A and80B, so that they are orthogonal to lines103L and103R. Thus the orientation of the combiners to their frames is given by equations (A).

For the second situation, where professional22is looking at location105, the directions of gaze of the professional are shown by lines105L and105R. These directions are respectively changed from the “straight ahead” directions by βL, βR. The values of βL, βRare given by equations (B):

βL=a⁢cos((w2)2+L2-R1+4⁢L2w2(R-w2)2+L2)⁢βR=a⁢cos((w2)2+L2+R1+4⁢L2w2(R+w2)2+L2)(B)

For the second situation processor26rotates combiners52A and52B, within their respective frames80A and80B, so that they are orthogonal to lines105L and105R. Thus the orientation of the combiners to their frames is given by equations (B), and these orientations are illustrated in the figure.

FIG.3Dis a graph of angles βL, βRvs. R for values of L=512 mm, w=60 mm.FIG.3Eis a graph of absolute angles γL, γR, of the angles made by combiners52A,52B with their frames where
γL′=βL+αL,γR=βR+αR(C)

From the above equations, as well as from the graphs, it is apparent that the angles made by combiners52A,52B with their respective frames are different, as professional26gazes at a region of interest. In addition, if the professional changes his/her gaze, the changes of the combiner angles to maintain orthogonality with the gaze directions are also different.

It will be understood that calculations based on equations herein, including equations (A), (B), and (C), assume that combiners52A,52B transmit rays that are orthogonal to the combiners. Those having ordinary skill in the art will be able to adapt the calculations, mutatis mutandis, for situations where the combiners transmit non-orthogonal rays.

FIG.4is a schematic diagram illustrating derivation of the dimensions of occlusion mask80, according to an embodiment of the present invention.FIG.4is derived from a section ofFIG.3B, specifically the section illustrating eye120A, with its pupil124A, being occluded by mask80A″ while the eye is gazing at ROI34.FIG.4illustrates a cross-section of the eye, the occlusion mask, and the region of interest. The figure has been drawn on xy axes with an origin O of the axes at the center of pupil124and the directions of the x and y axes being respectively orthogonal to and in the plane of the pupil. Mask80A″ and ROI34are assumed to be orthogonal to, and to be symmetrically disposed with respect to, the x-axis. Pupil124is assumed to be substantially circular. For simplicity, mask80A″ and ROI34are also assumed to be substantially circular. However, those having ordinary skill in the art will be able to adapt the following description, mutatis mutandis, for regions of interest and occlusion masks that are non-circular, so that the scope of the present invention is assumed to comprise both circular and non-circular regions of interests and masks.

The diagram has been drawn assuming that mask80A″ just completely occludes ROI34. Thus a ray HB, from an upper edge H of ROI34to an upper edge B of pupil124A touches an upper edge F of mask80A″. Similarly, a ray GA, from a lower edge G of ROI34to a lower edge A of pupil124A touches a lower edge E of mask80A″. Rays HB and GA are assumed to cross at an imaginary point J. A line from upper pupil edge B parallel to the x-axis cuts mask80A″ at K and ROI34at M.

In the description below:p is the apparent diameter of pupil124A, as measured externally to eye120A, corresponding to AB; andd is the diameter of mask80A″, corresponding to EF; d=d1for a realistic case of p>0, d=d0is the diameter of the mask for a theoretical “pinhole” case of p=0.

In addition,D is the diameter of ROI34(which is occluded by mask80A″), corresponding to GH;L is the distance from pupil124A to ROI34;l1is the distance from pupil124A to point J; andl is the distance from pupil124A to mask80A″.

InFIG.4ΔJFE∥ΔJHG, so that

dD=l+l1L+l1(1)

From equation (1),

d=l+l1L+l1·D(2)

If l1=0, (for the theoretical case of p=0), then
d=d0=1/L·D(3)

If l1>0, for the realistic case of p>0, then

d=d1=l+l1L+l1·D(4)Δ⁢BFK∥Δ⁢BHM,so⁢that⁢FKBK=FMBM(5)

For p>0 (so d=d1) and substituting values of d1, p, l, and L for FK, BK, FM, and BM in equation (5) gives:

d12-p2l=D2-p2L(6)

Equation (6) rearranges to:

d1=l⁡(D-p)L+p(7)

Equation (7) gives dimensions of mask80A″, i.e., its diameter d1, in terms of the diameter D of ROI34, the distance1of the mask from the pupil, the diameter of the pupil, and the distance L of the ROI from the pupil.

For typical values of 1=2 cm, L=50 cm, p=0.3 cm, and D=15 cm the diameter of mask80A″ to just give complete occlusion is, from equation (7), approximately 0.9 cm. For the same values but with p=0.15, the mask diameter is approximately 0.7 cm.

While, as described above, mask80A″ completely occludes ROI34, there are regions outside ROI34that are partly occluded by the mask. The partial occlusion follows from the finite, non-zero diameter of the pupil of eye, in the example described here pupil124A, and is described in more detail with reference toFIG.5below.

FIG.5is a schematic diagram illustrating partial occlusion of an area around ROI34, by mask80A″, according to an embodiment of the present invention.FIG.5is based uponFIG.4, but for clarity some of the elements ofFIG.4are omitted inFIG.5, while other elements are added in. Thus, a line through point A, parallel to the x-axis, cuts ROI34at N. A point Q, lying in the same plane as ROI34, and at a distance R from the x-axis, is assumed to project two rays—a lower ray150which touches lower edge A of the pupil, and an upper ray which, but for the presence of mask80A″, would touch upper edge B of the pupil. Point Q is thus partly occluded by mask80A″.

InFIG.5at a distance x1from the pupil lower ray150is assumed to be a distance f1(x1) from the x-axis, and upper ray160is assumed to be a distance f2(x1) from the x-axis. A line parallel to the y-axis, at x1, cuts BM at S, AN at T, upper ray160at V and lower ray150at W. Upper ray160cuts mask80A″ at V, and lower ray150cuts a plane containing the mask at W′.

At mask80A″ the distances of lower ray150and of upper ray160from the x-axis are respectively f1(1) and f2(1), and the width of the beam between the upper and lower rays is:
f1(1)−f2(1)  (8)

From the diagram,partial occlusion occurs if:
f1(1)>d/2 andf2(1)<d/2  (9)no occlusion occurs if:
f2(1)≥d/2  (10)and full occlusion, corresponding to the situation illustrated byFIG.4, occurs if:
f1(1)≤d/2  (11)

From expressions (8) and (9), and inspection ofFIG.5, an equation for the fraction F2Dof occlusion occurring is:

F2⁢D=d2-f⁢2⁢(l)f⁢1⁢(l)-f⁢2⁢(l)(12)

(The subscript 2D indicates that the fraction considered here is for the two-dimensional case illustrated inFIGS.4and5. A fraction for the three-dimensional case is referred to below.)

Since ΔATW∥ΔΔNQ

f⁢1⁢(l)=lL⁢(R-p2)+p2(13)

Since ΔBSV∥ΔBMQ

f⁢2⁢(l)=lL⁢(R+p2)-p2(14)

From equations (13 and (14) the diameter of the cone cross-section from Q at mask80A″, which is f1(1)-f2(1), is given by:
f1(1)−f2(1)=V′W′=p(1−1/L)  (15)

Substituting equations (14) and (15) into equation (12) gives the following expression for F2D:

F2⁢D=V⁢′⁢EV⁢′⁢W⁢′=d2-lL⁢(R+p2)+p2p⁡(1-lL)(16)

Inspection of equation (16) indicates that the fraction of occlusion at point Q is a function of pupil diameter p, and also decreases linearly as R increases.

HG is a cross-section of circular ROI34, so that it will be understood that GQ is a cross-section of a circular, partially occluded circular ring130surrounding ROI34. As illustrated inFIG.5, there is a point Q′, having the same distance R as Q from the x-axis (but on the opposite side of the axis), and in the same plane as ROI34, so that HQ′ is also a cross-section of ring130.

The rays from point Q define a cone of rays emanating from Q, and this cone cuts mask80A″ in a circle having a diameter V′W′, the diameter being given by equation (15). The cutting of mask80A″ by the cone of rays from Q is described with reference toFIG.6below.

FIG.6illustrates mask80A″ drawn in a plane orthogonal to the x-axis, according to an embodiment of the present invention. A circle170, which has a diameter given by equation (15), is the circle cutting the plane containing mask80A″. A portion174of this circle is occluded by circular mask80A″, and a portion178is transmitted. The expression for the two-dimensional fraction of occlusion F2Dgiven by equation (16) corresponds to analysis along a line FV′EW′.

There is a corresponding equation for a three-dimensional fraction of occlusion F3D, given by the following expression:

F3⁢D=AAL(17)where A is the area of portion174, andALis the area of circle170.

F3Dmay also be written as:

F3⁢D=AAL=α1⁢DL24+α2⁢d24-M⁢DL2⁢sin⁡(α1)π⁢DL24=1π⁢p2(1-lL)2⁢(a⁢cos⁡(p2(1-lL)2+4⁢(R⁢lL)2-d24⁢R⁢lL⁢p⁡(1-lL))p2⁢(1-lL)2+a⁢cos⁡(d2+4⁢(R⁢lL)2-p2(1-lL)24⁢R⁢lL⁢d)⁢d2-2⁢R⁢lL⁢p⁡(1-lL)⁢sin⁡(a⁢cos⁡(p2(1-lL)2+4⁢(R⁢lL)2-d24⁢R⁢lL⁢p⁡(1-lL))))(18)

From equation (18), F3Dis a function of pupil diameter p, and the equation provides numerical values of F3Dfor selected values of d, R, p, 1, and L.

FIG.7illustrates graphs of occlusion vs. distance, according to an embodiment of the present invention. The graphs have been drawn assuming the following values:L=50 cml=2 cmP=0.3 cmD=15 cm

From equation (7) the diameter of the occlusion mask to fully occlude an ROI with diameter D of 15 cm is d=0.888 cm. The graphs ofFIG.7have been drawn with d set at 0.894 cm.

From equation (15) the diameter of circle170is 0.288 cm, so that the value of the area ALof the circle is 0.065144 cm2.

A solid line graph200illustrates the full and partial occlusion vs. distance (from the center of the ROI) for the three-dimensional case comprising equation (18). The measurements of occlusion have been normalized, so that for an LCD screen a full occlusion of 95% is normalized to 1, and a full transparency (of 60% occlusion) is normalized to 0. A broken line graph204illustrates the full and partial occlusion vs. distance for the two-dimensional case comprising equation (16). As is apparent from both graphs, there is full occlusion, for a mask of diameter d=0.894 cm, for a region208up to approximately 8 cm from the center of the ROI, and partial occlusion in a region212from approximately 8 cm to approximately 15 cm. The fraction of partial occlusion decreases monotonically in region212.

FIG.8is a flowchart of steps performed in operation of system20, according to an embodiment of the present invention. The steps are assumed to be performed by processor26and, as necessary, professional22for use in a procedure on patient30performed by the professional using device38(FIG.1). In an initial calibration step300, frame54is adjusted to center combiners52with respect to the eyes of the professional. The sizes of the combiners and the distances of the combiners from the eyes of the professional are also measured and recorded by the processor (the use of the measurements is described further below). In step300assembly24is calibrated, i.e. optical elements of the assembly are registered with each other. Thus, devices68are oriented on frame54to capture generally similar images from the region in front of combiners52. If sensor72is present it is also aligned to capture a corresponding non-visible image from the region.

Processor26also orients the images from micro-projectors64, by registering the images projected by the micro-projectors onto combiners52with the scene viewed by professional22through the combiners. The registration may be accomplished by the professional viewing a scene through combiners52, together with an image of the same scene as it is captured by devices60and projected by the micro-projectors onto the combiners. The professional then adjusts the orientation of the micro-projectors and/or the capturing devices so that the projected image and the viewed scene coincide.

Typically the registration and adjustment of the micro-projectors and the capturing devices is performed for different regions of combiners52, such as the left and right peripheral regions, the upper and lower peripheral regions, and a central region. In addition, the registration and adjustment may be performed for different scenes according to the distance of the scene from the combiner, such as a scene of relatively near elements, typically up to 1 m from the combiner, and a scene of relatively far elements, typically greater than 1 m from the combiner. The registrations and adjustments of the micro-projectors and the capturing devices are typically different for the different regions of the combiners, as well as for scenes at different distances from the combiners. Processor26stores the different registration data acquired during the calibration step for use when the professional is using assembly24.

During the calibration step the sizes of the pupils of the eyes of professional22are measured. In one embodiment professional22gazes at a circular object of a known diameter and at a known distance from the professional, and processor26presents an occlusion mask on screens60to the professional. The professional then adjusts a diameter of the occlusion mask until complete occlusion of the object is achieved. As is apparent from equation (7), the diameter of the completely occluding mask provides a value for the pupil diameter, since d1, 1, L and D (terms in equation (7)) are all known.

Alternatively or additionally, the professional may look into a mirror while image capturing devices60acquire images of the reflected scene, in this case the professional wearing assembly24. Processor26analyzes the acquired images, by processes that are well known in the art, to identify the pupils of the professional as well as the outlines of combiners52. The processor then compares the diameters of the pupils with the known dimensions of the combiners, so as to determine values for the diameters.

The measurements of the pupil diameters are taken for different ambient light brightnesses, and the ambient brightness values may be determined from the signal levels of the images acquired by devices68. Processor26stores the values of the pupil diameters, and the corresponding brightness levels.

As stated above, processor26is configured to track device38, using the one or more identifying elements39(FIG.1). In calibration step300the processor initiates tracking of device38, and professional22confirms that the tracking is acceptable.

In an ROI defining step302, ROI acquisition marker35(FIG.1) is positioned on patient30, so as to define a region of interest of the patient selected by the professional, herein assumed to be ROI34. As explained above, marker elements36of marker35define the position of ROI34, and the size of the ROI may be defined by the professional. Typically there are at least three marker elements36, although more may be used, and characteristics of the elements, such as their color and/or shape, are selected so that they may be easily distinguished from patient30. If assembly24comprises sensor72with an infra-red projector, marker elements36may be configured as retro-reflectors which selectively reflect only infra-red radiation.

In an imaging step304, image capturing devices68acquire images of the scene in front of assembly24. Sensor72, if present, also captures a corresponding image of the scene. Processor26analyzes the images to identify marker elements36, and from the identified elements determines the orientation of ROI34with respect to assembly24, and also the distance of the ROI from the assembly. Even if sensor72is not present, it will be understood that having two devices68acquiring respective images of the scene simplifies the analysis needed to be performed by the processor to identify elements36. In addition, having two capturing devices68reduces the number of elements36required to accurately determine the orientation and distance of the ROI with respect to assembly24, compared to the number required if only one capturing device68is used. With two capturing devices68the inventors have found it is sufficient to have one marker with three marker elements to accurately locate the ROI with respect to assembly24. If sensor72is present, its image alone may be sufficient to identify elements36, although typically processor26uses the images from devices68to improve the accuracy of the orientation and distance measures of the ROI determined by the sensor.

Processor26also analyzes the images acquired by devices68in order to determine a measure of the brightness of the scene in front of assembly24.

In a frame orientation step305, the processor rotates combiners52A and52B with respect to their respective frames so that the combiners are orthogonal to the gaze directions of the professional towards the ROI. The processor uses equations (A), (B) and/or (C) to determine the angles of rotation of the combiners.

In a masking step306, the processor generates circular occlusion masks80in screens60. The processor, using the orientation of the ROI measured in step304and the central adjustment of combiners52in step300, determines positions for the masks that will occlude ROI34. From the brightness measured in step304, and from the correspondence between pupil size and brightness stored in initial step300, the processor estimates a value of the pupil diameter of the professional.

In one embodiment the processor sets the diameter of masks80according to equation (7), i.e., inter alia, according to the professional's pupil size, so that the masks fully occlude ROI34. In this case partially occluded ring130surrounds ROI34, the fraction of partial occlusion within the ring being given by equations (12) and (18).

In some embodiments the processor determines sections of the scene corresponding to partially occluded ring130, and as acquired by devices68. The processor then configures micro-projectors64to overlay video of the acquired sections onto the partially occluded ring, so as to compensate for the partial occlusion. Typically, processor configures the intensity of the projected video to be the inverse of the fraction of the occlusion.

In an alternative embodiment, rather than setting the diameter of the masks to be according to equation (7), the processor sets the diameter to be reduced from the value determined by the equation. The reduction is typically determined by professional22. In one embodiment the diameter is set to be 90% of the value determined by equation (7).

In a further alternative embodiment, the processor, using instructions from professional22, sets the diameter of the masks to be larger than the diameter of equation (7). In one embodiment the diameter is set to be 110% of the value determined by equation (7).

In a mask projection step308processor26uses micro-projectors64to project augmented video onto occlusion masks80. In the case of the augmented video including two or more types of images being projected onto the masks, processor26registers the images with each other. However, the images are not necessarily registered, and are typically misaligned, with the scene surrounding and outside the masks. Thus, as exemplified byFIG.2Eand the description of the figure, a video image114of the upper portion of the distal end of device38, together with a stored image112corresponding to the lower portion of the distal end, are registered together and are projected onto masks80. As is also illustrated inFIG.2E, the images on masks80are typically misaligned with the visible scene outside the masks.

In a further projection step310, processor26uses micro-projectors64to project augmented video onto the partially occluded ring surrounding the masks, and/or the non-occluded section of combiners52. As in step308, multiple image types are registered together, but are typically misaligned with the visible scene of the non-occluded section.

Typical images that are projected in steps308and310include, but are not limited to, those described above with respect toFIGS.2C,2D, and2E, and the choice and positioning of the images is typically under the overall control of professional22. In a mentoring situation, at least some of the images are typically under control of a mentor of professional22.

It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.