Cognitive displays

A cognitive mirror apparatus includes an imaging device configured to capture a plurality of images, a controller configured to determine a cognitive state of a user; and a cognitive mirror configured to display a virtual reflection comprising one or more of the plurality of images, wherein the virtual reflection of the cognitive mirror is determined using the cognitive state of a user.

BACKGROUND

The present disclosure relates to display technology, and more particularly to a display having changeable content and characteristics.

Driver aids are increasingly common in vehicles. These include information displays and notifications to improve the safety of the driver. These driver aids typically require a response by the user and are thus passive type devices.

BRIEF SUMMARY

According to an exemplary embodiment of the present invention, a cognitive mirror apparatus includes an imaging device configured to capture a plurality of images, a controller configured to determine a cognitive state of a user, and a cognitive mirror configured to display a virtual reflection comprising one or more of the plurality of images, wherein the virtual reflection is determined using the cognitive state of a user.

DETAILED DESCRIPTION

According to an exemplary embodiment of the present invention, a cognitive display or cognitive mirror (used herein interchangeably) apparatus is configured to adjust one or more view and/or image properties. The adjustments are selected to aid a user (e.g., a driver, security guard, hospital worker) in a differential manner depending on a cognitive state of the user. The cognitive mirror can be implemented as a stand-alone solution or to supplement another mirror (e.g., a traditional side-view mirror used on a truck).

In one or more embodiments, a cognitive mirror apparatus100includes a cognitive mirror101and a video imaging device102. The cognitive mirror displays a view captured by the video-imaging device (e.g., a video camera). The view may not be visible by the user using traditional mirrors (e.g., rearward of a vehicle).

According to an exemplary embodiment of the present invention, an apparatus generates an image of one or more areas around (e.g., to the side and rear of) a vehicle. With reference toFIG. 2andFIG. 3, the cognitive mirror apparatus300includes a weather resistance cognitive mirror201located outside of the vehicle (e.g., on a lower portion of a side mirror). In this example, a first camera202is located on the rear of a trailer in a center mounted position having a first view203, while a second camera204(implied by not explicitly shown) is located on a passenger side near the front of the vehicle having a second view205so that a view of the entire side of the vehicle can be generated in the cognitive mirror201. In one or more embodiments, the cameras can be equipped with fixed wide-angle lenses.

A video signal from each camera (202/204) is transmitted to a switch/controller301. The switch/controller301controls an image displayed by the cognitive mirror201, for example to display an image corresponding to each signal for a pre-determined time or allow for a manual selection of any one of the cameras. The switch/controller301can determine a cognitive state of a user. The cognitive state can be determined using sensors for evaluating different parameters including eye movement, facial expression, posture, pulse, etc. Examples of the cognitive state include, for example, a normal state associated with one or more parameters in a predetermined normal range (e.g., normal pulse and eye movement), a preparative state associated with a shifting gaze in a particular direction such as in preparing to change lanes, etc. In such a manner the driver can look at the cognitive mirror201in order to see one or more additional outward views of the vehicle depending on his cognitive state, increasing safety and reducing a need to look in various directions to check for obstructions while backing up or changing lanes. For example, the image displayed by the cognitive mirror201can correspond to a preparative state associated gaze direction.

According to an exemplary embodiment of the present invention, the switch/controller301determines parameters related to the user. In the example of a driver, these parameters can include general parameters such as a measure of driving ability, average level of distraction, etc., and specific parameters including the length of the current trip, weather, etc. The switch/controller301can adjust display parameters using the parameters of the driver.

The parameters of the user can be determined from the camera's202/204and other sensors304(e.g., a speedometer of the vehicle, heart rate of the user captured from a user's heart rate monitor, etc.).

In addition, the parameters of the user may be stored and retrieved from a remote repository302or cognitive modeling facility303, for example, a driver's smart phone or activity monitor, or a cloud based cognitive interface management facility. In one or more embodiments of the present invention, the parameters of the driver are translated into one or more display parameters using a configurable mapping. The configurable mapping can be provided by the vehicle manufacturer, a regulatory body, a cognitive modeling and management service, the user, etc. In one example, the mapping relates a parameter of the driver (e.g., a level of driver distraction) to a shape of the cognitive mirror (e.g., convexity/concavity), either in a virtual or physical implementation. In another example, the parameter of the driver can include a level of driver reaction to backlighting/highlighting of objects in the cognitive mirror or an age of the driver, which can be used by the switch/controller301to control an angle of the cognitive mirror. In yet another example, a driver's driving experience is mapped to cognitive mirror attentional signals, such as a flashing alert when the cognitive mirror should be checked.

Adjustments in the display parameters can be made iteratively, sampling the user's cognitive state to convergence (e.g., on a threshold for confidence). For example, a cognitive mirror can adjust to a driver's point of view by iteratively making adjustments in the display parameters until the driver is judged to respond appropriately to surrounding vehicles or traffic (e.g., indicating that the cognitive mirror has reduced any blind spot in the driver's view). The determination of driver seeing the surrounding vehicles may be ascertained by analysis of driving behavior, looking back behavior, and eye movements. In one example, a virtual reflection displayed by the cognitive mirror can be adjusted for dazzle control based on an assessment of the cognitive state of the user (i.e., determine whether the user is squinting in a predominately low light environment as an indication of the cognitive state of being blinded by lights). In this case, the predominately low light environment can be determined based on time of day and the date (i.e., whether it is after dusk).

In one exemplary implementations, in the case of a parked vehicle, the cognitive mirror can display a zoomed image of the driver, depending on for example, the proximity of the driver to the mirror. In one example, the cognitive mirror apparatus can determine that the driver is moving towards the cognitive mirror and as a result display a zoomed image of the driver. In this case the cognitive mirror apparatus perceives a need of the driver (e.g., a close up view), and displays a zoomed view. In another exemplary implementation, the cognitive mirror can display a calendar, and as the user draws neared the cognitive mirror, the display can zoom on an area of the calendar determined to be of interest to the user, e.g., based on eye movement.

The cognitive mirror apparatus can learn based on aggregate experience of many users in relation to correlations among properties, captured incidents, frequency of activity, etc.

In another embodiment, the content and characteristics of the cognitive mirror change depending on perceived cognitive states of individuals in the vicinity of the user, for example, based on a level of traffic congestion. These individuals can include, for example, cyclists, pedestrians, motorcycles, drivers of cars, etc. located around the user's vehicle. In this embodiment, the content and characteristics of the cognitive mirror can include a demarcation of vehicle within a radius R, e.g., by an outline around the vehicle in the display of the cognitive mirror, a change in brightness of the cognitive mirror, an arrow pointing to the vehicle in the display of the cognitive mirror, a change in view or perspective that makes the vehicle more apparent in the display of the cognitive mirror, etc. The demarcation of a vehicle in a specific region of the cognitive mirror includes an estimation based on a driver's eye movement, the angle of the cognitive mirror, and the location of an object (e.g., the vehicle being marked) imaged. For example,FIG. 5shows an illustration of a cognitive mirror and virtual reflection500according to an exemplary embodiment of the present invention in which an attentional signal501(e.g., a flashing alert) is displayed, together with a demarcation502of a vehicle503captured by a video-imaging device and displayed as in the virtual reflection500.

The cognitive state of the individuals in the vicinity of the user can be estimated with confidence C according to, for example, perceived driving patterns, vehicle type, perceived distraction level, biometric inputs (e.g., information gained from other drivers, such as state of attentiveness of their driving based on following their eyes using eye trackers, clenching of the steering wheel, volume of noise in cars, a tendency to drift from the road with an attempt to recover, other indicators of fatigue or anger, etc.), arousal state (e.g., distractions caused by additional occupants in a nearby vehicle), real-time video of another driver's face and eyes, etc. This video can be anonymized so that the video remains private.

According to an exemplary embodiment of the present invention, a communication device305(seeFIG. 3) of the cognitive mirror apparatus300shares driver cognitive states among cognitive mirror apparatus in a vicinity of the user's vehicle (e.g., within about 150 feet of the user's vehicle or within about 5 seconds of the user's vehicle given vector data) implemented in vehicles on the road. The sharing of cognitive states through communication devices (e.g.,305) facilitates identification in a cognitive mirror of a vehicle driven by a driver in a particular cognitive state. A demarcation can then include information about the cognitive state of an operator of a vehicle imaged in the cognitive mirror. In one or more embodiments, the cognitive mirror can share video information with other vehicles on the road. For example, a PIP (picture in picture) display can show a view obtained from a vehicle directly behind the user's vehicle. In this way, information about more distant traffic conditions (e.g., obstructed by other vehicles) can be made available to the driver. PIP's can be recursive to allow additional views of even more distant road conditions.

According to one or more embodiments of the present invention, the communication device305is a component of a network connecting a plurality of cognitive mirror apparatus in the vicinity of the user's vehicle. In one example, the network is configured as with central or star data flow topology, with the communication device305functioning as a hub. In another example the network is configured as a peer-to-peer network, with the communication device305function as a peer.

According to an exemplary embodiment of the present invention, video content is stored. Stored video content can be played back at a later time, for example, for educational purposes.

In one or more exemplary embodiments, a cognitive mirror apparatus is implemented on vehicles including aircraft, motorcycles or bicycles, in indoor applications such as hospitals and secure environments wherein dome or drop-in cognitive mirrors can draw a user's attention (e.g., by a visual attentional signal or alert notification) to an approaching object around a corner, possibly based on a perceived speed of the object so as to avoid collisions. In another example embodiment, a virtual cognitive mirror can be implemented in an Augmented Reality (AR) environment. The virtual cognitive mirror can be useful in medical devices that currently make use of mirrors, including dental mirrors, speculums, and certain cognitive tests.

In one or more exemplary embodiments, a cognitive mirror apparatus is implemented in a virtual environment such as a video game, tutorial, training simulator, etc. For example, a cognitive mirror in a virtual vehicle can change contents and characteristics depending on the cognitive state of user.

By way of recapitulation, according to an exemplary embodiment of the present invention,FIG. 6shows a method600for operating a cognitive mirror apparatus including capturing input data601including video signals from one or more input devices (e.g., cameras, sensors, etc.), determining parameters of the cognitive mirror602based on the captured input data (e.g., using a cognitive state of a user), adjusting content and/or characteristics of the cognitive mirror603(e.g., changing an alignment, adding attentional information, zooming, etc.) and displaying a virtual reflection comprising an image corresponding to the adjusted content and/or characteristics on the cognitive mirror604. The adjustments can include changes in the content and characteristics of the cognitive mirror, including convexity/concavity, alignment, brightness, contrast, headlight dazzle control, input from displayed video cameras, etc. The cognitive/context parameters can include context of area around a vehicle, the user's sightlines, speed of the vehicle, characteristics of the vehicle, characteristics of the user (e.g., whether or not the driver is wearing sunglasses, pupil dilation, tiredness, and distraction), etc.

The methodologies of embodiments of the disclosure may be particularly well-suited for use in an electronic device or alternative system. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “processor,” “circuit,” “module” or “system.”

Furthermore, it should be noted that any of the methods described herein can include an additional step of providing a cognitive mirror apparatus including a cognitive mirror (e.g.,201), a switch/controller301, and input devices (e.g., a camera202and sensor304). Further, a computer program product can include a tangible computer-readable recordable storage medium with code adapted to be executed to carry out one or more method steps described herein, including the provision of the system with the distinct software modules.

Referring toFIG. 7;FIG. 7is a block diagram depicting an exemplary computer system embodying a cognitive mirror apparatus300(seeFIG. 3) according to an embodiment of the present invention. The computer system shown inFIG. 7includes a processor701, memory702, display703, input device704(e.g., keyboard), a network interface (I/F)705, a media IF706, and media707, such as a signal source, e.g., camera, Hard Drive (HD), external memory device, etc.

In different applications, some of the components shown inFIG. 7can be omitted. The whole system shown inFIG. 7is controlled by computer readable instructions, which are generally stored in the media707. The software can be downloaded from a network (not shown in the figures), stored in the media707. Alternatively, software downloaded from a network can be loaded into the memory702and executed by the processor701so as to complete the function determined by the software.

The processor701may be configured to perform one or more methodologies described in the present disclosure, illustrative embodiments of which are shown in the above figures and described herein. Embodiments of the present invention can be implemented as a routine that is stored in memory702and executed by the processor701to process the signal from the media707. As such, the computer system is a general-purpose computer system that becomes a specific purpose computer system when executing routines of the present disclosure.

Although the computer system described inFIG. 7can support methods according to the present disclosure, this system is only one example of a computer system. Those skilled of the art should understand that other computer system designs can be used to implement embodiments of the present invention.