Patent Publication Number: US-2023164286-A1

Title: Methods and Apparatuses for Compensating for Retinitis Pigmentosa

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation application of U.S. patent application Ser. No. 16/503,098, entitled “Methods and Apparatuses for Compensating for Retinitis Pigmentosa” and filed on Jul. 3, 2019, which claims the benefit of U.S. Provisional Application No. 62/694,173, entitled “Compensating for Retinitis Pigmentosa: Using dynamic shifts in field of view and magnification” and filed Jul. 5, 2018, the contents of both of which are hereby incorporated by reference in their entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The present invention generally relates to a vision-enhancement system and methods, and, more particularly, to a head-mounted and user-controllable method and system for vision-enhancement and a system and method for configuring a vision-enhancement system to compensate for those suffering from retinitis pigmentosa. 
     Discussion of the Background 
     The normal human visual system establishes a non-magnified representation of a scene in the visual periphery with a high-resolution representation at the center of the visual field. Thus, the cornea and lens of the eye focuses a viewed scene onto the retina, which includes the fovea near the center of vision, and a peripheral area. The fovea is a small area composed of closely packed cones near the center of the  Macula lutea  of the retina. The fovea is responsible for sharp central vision which is necessary for activities where visual detail is of primary importance, such as reading and driving. 
     The fovea is greatly expanded at the visual cortex and represents a significant magnification mechanism that allows a normally sighted person to discern the region of the visual world that is in “focus” on (faces, titles of dish soap, text), but sees that region in the broad context of an overall visual field. 
     People with retinitis pigmentosa (referred to herein as “RP”) suffer from “tunnel vision” with an accompanying reduction in the ability to capture enough light in low light situations (i.e., dusk, nighttime, theaters, and the like) to be able to navigate safely. These people retain central vision but lose most peripheral vision. In normal daylight situations, the effect of RP is to easily swamp the vision during normal sunlight to the point that the visual scene is obscured. The visual field can be as narrow as 1 degree (the width of your thumb at arm&#39;s length). Such people have difficulty with mobile activities such as navigating and recognizing both people and objects at a distance and can have difficulty reading. These disabilities greatly diminish their quality of life, greatly limiting their ability to socialize, shop, cook and travel. 
     Traditionally, the effects or RP may be compensated using an optical or digital “reverse telescope.” This solution, however, is inadequate because the magnification, field and focus point is fixed. The person is restricted to using the same parameters for all situations. This limits the utility of the device and its value to those suffering from RP. 
     Thus, there is a need in the art for a method and apparatus that permits for enhanced representation of the visual world that also enables mobility for navigation and recognition. The methods and apparatus should be easy to use, allow the user to control what is being viewed, and be inexpensive. 
     BRIEF SUMMARY OF THE INVENTION 
     The present invention overcomes the limitations and disadvantages of prior art vision-enhancement systems and methods by providing the user with a head-mounted, user-adjustable field of view system that provides enhancements for various low vision problems. 
     It is one aspect to provide a device that can present to the user, in real-time, an enhanced video image of a scene surrounding the user so that the user can perform routine tasks. Certain embodiments include: 1) a head-mounted video camera that can obtain an input stream of video images of the scene in front of the user&#39;s head; 2) a processor to process the stream of input video images into a stream of output images that corrects for the user&#39;s vision deficiencies; and 3) a display for the processed video images that is viewable by the user. In certain embodiments, the system presents an image of adjustable size and presents a scene within a smaller field of view than what would be viewed with normal vision. 
     It is another aspect to provide a wearable and portable vision-enhancement system for persons suffering from RP. The system includes a memory including a stored program; a camera mounted on the user aimed to view the scene in front of the user&#39;s head and operable to obtain input video images of the scene; a processor programmed to execute the stored program to transform the input video images into a stream of output video images. In one embodiment, the transformation from input video images results in output video images that occupy a smaller field of view than uncorrected vision. In effect, the system provides a zoomed-out version of the input images similar to looking through the “wrong” end of a telescope. The zoomed-out images are referred to herein as being “minified,” which is the opposite of being magnified. This minified view gives the patient a smaller field of view that can be “scanned” with the patient&#39;s limited (often 1 degree) visual field. The program is also operable in response to a command by the patient after viewing the minified contextual scene to magnify within the smaller FOV so as to focus in on a particular feature of the scene (another person&#39;s face, a label on a package, title of a book, etc.). 
     It is one aspect to provide a portable vision-enhancement system wearable by a user. The system includes: a memory including a stored program; a camera mounted on the user aimed to view the scene in front of the user and operable to obtain input video images of the scene; a processor programmed to execute the stored program to change the input video images into a stream of output video images; a screen disposed to display the output video images for viewing by the user; and a user-operable controller for generating an input to the processor by which the size of the output video images are adjusted. 
     It is another aspect to provide a method for compensating for retinitis pigmentosa for a user using a system including a camera, a processor, and a screen. The method includes: accepting a stream of video images from the camera; processing the accepted stream of video images in the processor into a stream of output video images; and accepting an input from the user to adjust the size of the output video images on the screen. 
     These features, together with the various ancillary provisions and features which will become apparent to those skilled in the art from the following detailed description, are attained by the vision-enhancement system and method of the present invention, preferred embodiments thereof being shown with reference to the accompanying drawings, by way of example only, wherein: 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
         FIGS.  1 A,  1 B, and  1 C  show a first embodiment user-controllable vision-enhancement system, where  FIG.  1 A  shows the system on a user,  FIG.  1 B  shows a smartphone used in the system and  FIG.  1 C  shows the body of the goggle used in the system; 
         FIG.  1 D  shows a system which may be used to configure a vision-enhancement system; 
         FIG.  2    shows an output image of a scene as would be presented to a user prior to being minified; 
         FIG.  3    shows a minified output image of the scene  FIG.  2   ; and 
         FIG.  4    shows a magnified minified output image of the scene of  FIG.  3   . 
     
    
    
     Reference symbols are used in the figures to indicate certain components, aspects or features shown therein, with reference symbols common to more than one figure indicating like components, aspects or features shown therein. 
     DETAILED DESCRIPTION OF THE INVENTION 
     Certain embodiments of the inventive vision-enhancement system described herein include: 1) a video camera to capture the scene that would be in the user&#39;s field of view if the user was not wearing the system; 2) a processor to image process the video of the captured scene; and 3) a screen to present the processed video images to the user&#39;s eyes. The modified video image compensates for the user&#39;s particular vision problem, such as tunnel vision, by providing more of the central image of interest to the more active, less impacted portions of the user&#39;s eyes, while retaining the peripheral context of the scene. Thus, the image of the of the scene is presented to the user according to an image transformation that maps the captured image, pixel-by-pixel, to an image that presents a smaller field of view than if it were viewed without the system and is thus more useful to the user. 
     In one embodiment, minified images are presented to the user of the vision-enhancement system. The term “minified” is used to denote an output image that is a zoomed out version of the input images. Thus, for example, a scene that would be viewed without enhancement over a 70-degree field of view is reduced in size so that the same scene is viewed within a smaller field of view, such as, for example and without limitation, a 35 degrees. A minified image is similar to the effect of looking through the “wrong” end of a telescope and is the opposite of being magnified. This minified view gives the patient suffering from RP a contextual view of the scene. The image-processed video modifies a scene in a way that allows the user to more easily recognize people or identify items in their surroundings. Thus, the user may be able to identify items on the supermarket shelf, recognize friends and neighbors, read street signs, and sit in an armchair to read a book. In other embodiments, the user is also able to magnify or zoom in to the image to allow greater detail to be discerned. 
     By way of a specific embodiment,  FIGS.  1 A,  1 B, and  1 C  show a first embodiment user-controllable vision-enhancement system  100 , where  FIG.  1 A  shows the system on a user,  FIG.  1 B  shows a smartphone used in the system and  FIG.  1 C  shows the body of the goggle used in the system. System  100  includes a smartphone  110  supported in headset  120 . Smartphone  110  includes the electronics necessary for the vision-enhancement system  100 , including a processor and memory (not shown), a forward-facing camera  111 , as shown in  FIG.  1 A , and a screen  113  on the side opposite the camera, as shown in  FIG.  1 B . Smartphone  110  also includes an electrical connector  117  and may also include a backward-facing camera  115 , which may be used in certain embodiments. As described subsequently, processed camera images are displayed on one portion of screen  113  shown as a left area  112  and a second portion of the screen shown as a right area  114 . 
     Headset  120  include a body  122  and a strap  125  for holding the headset on the user&#39;s head and a connector  128  that mates with smartphone connector  117 . Body  122  includes, as shown in  FIG.  1 A , a pair of clamps  121  for removably restraining smartphone  110  and making the electrical connection between connectors  117  and  128 , and input device  123  for providing input to the smartphone through the connectors and, as shown in  FIG.  1 C , a left lens  124  and right lens  126  and a focusing wheel  127 . When assembled as in  FIG.  1 A , with smartphone  110  held in place by clamps  121 , system  100  presents what is displayed in area  112  of screen  113 , through lens  124 , to the user&#39;s left eye, and what is displayed in area  114  of the screen, through lens  126 , to the user&#39;s right eye. The user may use focusing wheel  127  to adjust the focus. In certain embodiments, headset  120  are adapted to accept user input from input device  123 , which may control or otherwise provide inputs to the accepted smartphone  110 . 
     In certain embodiments, smartphone  110  is provided with programming, as through a vision-enhancement application (referred to herein as a “VE App”) which can: 1) operate camera  111  in a video mode to capture a stream of “input images”; 2) perform image processing on each input image to generate a stream of “output images”; and 3) present the stream of output images to screen  113 . In certain embodiments, each of the stream of output images is presented sequentially side-by-side as two identical images—one in area  112  and one in area  114 . Further, it is preferred that vision-enhancement system  100  operate so that the time delay between when the input images are obtained and when the output images are provided to screen  113  be as short as possible so that a user may safely walk and interact with the environment with headset  120  covering their eyes. 
     In certain embodiments, the VE App may also provide a menu of options that allow for the modification of how vision-enhancement system  100  generates an output image from an input image. Thus, for example, vision-enhancement system  100  may execute image-processing algorithms having parameters, where the parameters are changeable through the menu by, for example, setting parameter values for magnification, or the size and shape of magnification of the output image. 
     Vision-enhancement system  100  has adjustable features that allow it to match the physiology of the user for use in different settings. These features are generally set once for each user, possibly with the need for periodic adjustment. Thus, for example, given the spacing between screen  113  and the eyes of user U, focusing wheel  127  permits for an optimal setting of the distance to lenses  124  and  126 . In addition, lenses  124  and/or  126  may include refractive error correction. Further, it is important that the viewed spacing between the images in areas  112  and  114  match the user&#39;s interpupillary distance (IPD). This may be accounted for, by example, by shifting the spacing of the output images in areas  112  and  114  to match the IPD. 
     In various embodiments, the user may adjust settings using input device  123 , which may be a touchpad, and which is electrically connected to smartphone  110 , which is further programmed to modify the VE App according to such inputs; a Bluetooth game controller that communicates with the smartphone  110  via Bluetooth; voice control using the microphone of the phone; or gesture control using available devices such as the NOD gesture control ring (see, for example, http://techcrunch.com/2014/04/29/nod-Bluetooth-gesture-control-ring/). 
     In addition, there are other features of vision-enhancement system  100  that can either be set up once for a user or may be user-adjustable. These features may include, but are not limited to, adjustments to the magnitude, shape, size, or placement of minified or magnified portions of the output image, and color enhancement functions such as contrast, blur, ambient light level or edge enhancement of the entire image or portions of the image. In other embodiments, the compass and/or accelerometers within smartphone  110  may be used for enhancing orientation, location, or positioning of output images. 
     In certain embodiments, sound and/or vibration may be provided on smartphone  110  to generate for proximity and hazard cues. In other embodiments, the microphone of smartphone  110  can be used to enter voice commands to modify the VE App. In certain other embodiments, image stabilization features or programming of smartphone  110  are used to generate output images. 
     In one embodiment, by way of example only, headset  120  are commercially available virtual-reality headset, such as Samsung Gear VR (Samsung Electronics Co. Ltd., Ridgefield Park, N.J.) and smartphone  110  is a Galaxy Note 4 (Samsung Electronics Co. Ltd., Ridgefield Park, N.J.). The Samsung Gear VR includes a micro USB to provide an electrical connection to the Galaxy Note 4 and has as input devices  123  a touch pad and buttons. 
     It will be understood by those in the field that vision-enhancement system  100  may, instead of including a combination of smartphone and headset, be formed from a single device which includes one or more cameras, a processor, display device, and lenses that provide an image to each eye of the user. In an alternative embodiment, some of the components are head-mounted and the other components are in communication with the head-mounted components using wired or wireless communication. Thus, for example, the screen and, optionally the camera, may be head-mounted, while the processor communicates with the screen and camera using wired or wireless communication. 
     Further, it will be understood that other combinations of elements may form the vision-enhancement system  100 . Thus, an electronic device which is not a smartphone, but which has a processor, memory, camera, and display may be mounted in headset  120 . Alternatively, some of the electronic features described as being included in smartphone  110  may be included in headset  120 , such as the display or communications capabilities. Further, the input control provided by input device  123  may be provided by a remote-control unit that is in communication with smartphone  110 . 
       FIG.  1 D  illustrates, without limitation, one embodiment of a clinical setup  140  that a clinician may user to configure vision-enhancement system  100 . Clinical setup  140  may allow a clinician to determine and setup the VE App by setting an IPD, the field of view (fov), background dimming, ambient light level, as well as parameters that are also user-adjustable, such as the size, shape, magnification, and location of enhanced vision features. 
     Clinical setup  140  thus allows for the adjustment or parameters within or used by the VE App that smartphone  110  runs to implement the vision-enhancement system  100 . Clinical setup  140  includes a monitor  142 , a Wi-Fi device  144  to allow screen  113  of smartphone  110  to be displayed on the monitor, and a Bluetooth controller  146  to communicate via Bluetooth with smartphone  110 . In general, clinical setup  140  accepts a video output from smartphone  110  of display  113  and projects what the user would see when using vision-enhancement system  110  on monitor  142 . 
     In certain embodiments, features or aspects of the inventive vision-enhancement system  100  may be adjusted by a clinician using clinical setup  140 . Using the setting up vision-enhancement system  100 , screen  113  of smartphone  110  is mirrored on a monitor using Wi-Fi device  144 , for example, so that the clinician can view what the user is viewing in vision-enhancement system  100 . The VE App on smartphone  110  includes a menu that allows for the selection of certain parameters that operate vision-enhancement system  100 . 
     The clinician has access to the commands in the menu of the VE App via remote Bluetooth controller  146 . In this way, the clinician can “tune” the device to the specific visual demands of the user. 
     In certain embodiments, Wi-Fi device  144  can be used to remotely add, augment or modify functions to allow vision-enhancements, mirror the display, monitor and control VE App configurations in a clinical environment. In certain embodiments, Bluetooth controller  146  can be used to control or modify visual enhancement functions. In certain other embodiments, the VE App may be reconfigured in a purely magnified format, making it possible for the low vision user to place phone calls, utilize maps, read announcements and perform all visual functions currently available to those with normal vision. 
     Examples of functional control of vision-enhancement system  100  provided by clinical setup  140  through the operation of the VE App may include but are not limited to: 1) mirroring the user&#39;s display via Wi-Fi; and 2) controlling the minification and/or magnification of the output image. The clinician will have the facility to control the low vision parameters such as amount and placement of minification or magnification, contrast, edge enhancement ambient level and other visual enhancements to customize the device for the user. 
     To better appreciate how the apparatus and methods described herein alter images for viewing,  FIG.  2    presents an example of what a vision-enhancement system  100  displays without the VE App running—that is, what would be normally shown on the screen prior to minifying the image. 
       FIG.  2    illustrates an image  200  in area  112  which may be presented to one or both of display areas  112 / 114  prior to the alteration of the image as described herein. In general, image  200  is representative of any image or portion of an image captured by camera  111  and stored in the memory of smartphone  110 . Image  200  includes an output image Sofa scene as the image of the scene prior to minification. Typically, an output image S occupies approximately 70 degrees when viewed by the user. That is, when area  112  is placed for viewing, the image presented subtends 70 degrees of the user&#39;s field of view. 
     For reference,  FIG.  2    shows two portions of area  112  that are discussed in greater detail with reference to  FIGS.  3  and  4   . These portions are: 1) the outline of an output area  203  of area  112  into which the image S will modified for presentation; and 2) a circle representing a user&#39;s visual field  203  on area  112 . The size of output area  203  is selectable by the user, the size of visual field  201  depends on the vision of the user, and the location of the visual field on area  112  depends on the gaze of the user—that is, it follows the movement of their eyes relative to the area. 
       FIG.  3    illustrates a minified output image of the scene  FIG.  2   . In an illustrative example, VE App of the vision-enhancement system  100  is operated to place output images within output area  203  and is also programmed to accept input through input device  123 , as discussed above, to control the size of output area  203  of display areas  112 / 114 . 
     Thus, for example, one embodiment of system  100  performs a transformation or mapping of the output image S into output image  300 , as shown in  FIG.  3    and which may be, for example, one or both of display areas  112 / 114 . In image  300 , the user has manipulated input device  123  to select the size of output area  203 . The unaltered output image, such as scene S, is reduced in size to fit within output area  203  and thus present a minified output image S′, which is, in effect, output image S reduced in size and bounded by the output area  203 . Due to the reduced area, an area B between output area  203  and area  112  does not contain the scene and may, for example, be colored black. 
     It will be appreciated that since display areas  112 / 114  are located at some specific distance from the user&#39;s eyes, output area  203  will thus occupy a visual field in proportion to the size of the output area. In one embodiment, output area  203  is smaller than area  112  and occupies a field of view that is less than or equal to 45 degrees, or, alternatively, to a field of view that is less than or equal to 35 degrees field. Thus, for example, output area  203  may occupy between 15 and 30 degrees when viewed by the user or may occupy 15 degrees, 20 degrees, 25 degrees, or 30 degrees. 
       FIG.  3    also shows, for reference, the user&#39;s visual field  201 . The actual location of visual field  201  depends on where the user is looking. Thus, visual field  201  will move about output area  203  as the user directs their gaze to different portions of output area  203 . As is apparent from comparing  FIGS.  2  and  3   , when output image S is minified as output image S′, the angular size of the image is decreased and more of the original output image S falls within image user&#39;s visual field  201 . 
     It will also be appreciated that output area  203  may be rectangular, as in the above embodiments, or may have a circular shape, or may be placed anywhere within image  300  as is appropriately tailored to a specific user&#39;s vision needs. 
     In addition to minifying the image, alternative embodiments include vision-enhancement system  100  programmed to enhance the light available in low light situations and reduce the intensity of light in bright light situations in response to the user&#39;s command. 
       FIG.  4    shows a magnified minified output image of the scene of  FIG.  3   . In an illustrative example, VE App of the vision-enhancement system  100  is programmed to accept input through input device  123 , as discussed above, to control the scale of output area  203  of display areas  112 / 114 . 
     Thus, for example,  FIG.  4   , shows an image  400  which includes an output image S″ within output area  203 , where the output area is determined as described above with reference to  FIG.  3   . Output image S″ is a magnified view of minified image S′—that is, S″ is a magnified minified image. In certain embodiments, the magnification of the minified image is modified by the user through input device  123 , as discussed above. In other words, the user may adjust the magnification of the minified image. 
     As is apparent from comparing  FIGS.  3  and  4   , magnifying a minified image permits the user wearing system  100  to see greater detail, which, in turn, facilitates reading and other activities. 
     In one embodiment, a VE App is provided to smartphone  110  which performs image processing on each image of the stream of video images and then presents each processed area simultaneously to areas  112  and  114 . VE App may perform the following steps: 
     1) start the method, which runs continuously until stopped by the user. 
     2) capture images the scene. Images from stream of images are read from camera into memory. Depending on the architecture of the particular device and the capabilities and performance of the camera and the processing CPU, this may be done in one of two ways: a) read the entire image into memory and then process all pixels in the image, or b) read a collection of pixels (essentially a slice of the image) and process it while another slide is being read. 
     3) determine the size of the image output area  203 . This may be preset in the VE App or may be controllable through input device  123 . 
     4) minify the captured image to fit within output area  203  and optionally blacken the area between output area  203  and the edge of areas  112 / 114 . 
     5) accept input from input device  123  to magnify the minified image. 
     Among the attributes of this embodiment are:
         1. Dynamic control by the user of minification size of the field of view.   2. Dynamic control by the user of magnification within the field of view.   3. User control of these parameters that provides the patient with the power to control the incoming scene. These parameters will change depending upon what is being viewed, ranging from faces, to text, to theater events. The auto-focus camera assures that image will be in focus at all distances.   4. Use of the camera/display system to compensate for low light conditions, making the images bright enough for an RP patient to see in twilight.   5. Enclosure of the entire scene in a headset, eliminating excessive light and glare that are particularly troubling for the RP patient.       

     Each of the embodiments of the devices and methods described herein is in the form of a computer program that executes as an app on a smartphone. It will be appreciated by those skilled in the art, embodiments of the present invention may be embodied in a special purpose apparatus, such as a headset which contain the camera, processor, and screen, or some combination of elements that are in communication and which, together, operate as the embodiments described. 
     It will be understood that the steps of methods discussed are performed in one embodiment by an appropriate processor (or processors) of a processing (i.e., computer) system, electronic device, or smartphone, executing instructions (code segments) stored in storage. It will also be understood that the invention is not limited to any particular implementation or programming technique and that the invention may be implemented using any appropriate techniques for implementing the functionality described herein. The invention is not limited to any particular programming language or operating system. 
     Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner as would be apparent to one of ordinary skill in the art from this disclosure in one or more embodiments. 
     Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention. 
     Thus, while there has been described what is believed to be the preferred embodiments of the invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention and it is intended to claim all such changes and modifications as fall within the scope of the invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present invention.