Smart necklace with stereo vision and onboard processing

A wearable neck device and a method of operating the wearable neck device are provided for outputting optical character recognition information to a user. The wearable neck device has at least one camera, and a memory storing optical character or image recognition processing data. A processor detects a document in the surrounding environment and adjusts the field of view of the at least one camera such that the detected document is within the adjusted field of view. The processor analyzes the image data within the adjusted field of view using the optical character or image recognition processing data. The processor determines output data based on the analyzed image data. A speaker of the wearable neck device provides audio information to the user based on the output data.

BACKGROUND

Field

The present disclosure relates to a wearable device, which provides haptic and audio feedback based on stereo camera input.

Description of the Related Art

Static and passive recorders have been provided for recognizing characters on a document. Users have to manually identify the documents themselves and place portions of a document in front of a camera of the static and passive recorder. Certain users, such as persons with visual impairments, desire additional assistance in locating a document and additional dynamic feedback relating to identified documents. Static and passive recorders known in the art do not dynamically recognize documents and do not dynamically provide helpful information about the documents. Furthermore, static and passive records known in the art do not have enhanced user-interactive features.

Thus, there is a need for an unobtrusive device which dynamically assists a user in identifying documents and interactively provides helpful information about the document.

SUMMARY

A wearable neck device and a method of operating the wearable neck device are provided for outputting optical character or image recognition information to a user. The wearable neck device has one or more cameras, and a memory storing optical character recognition processing data. A processor detects a document in the surrounding environment. The processor adjusts the field of view of the one or more cameras to be focused downwards such that the detected document is within the adjusted field of view. The processor analyzes the image data within the adjusted field of view using the optical character recognition processing data. The processor determines output data based on the analyzed image data. A speaker of the wearable neck device provides audio information to the user based on the output data.

The one or more cameras may include a first camera positioned at a first end of the band, and a second camera positioned at the second end of the band. The first camera and the second camera form a pair of stereo cameras configured to operate in conjunction to capture the image data. Alternatively, the one or more cameras may include an omni-directional camera or a wide-angle camera. The processor may re-adjust the field of view of the one or more cameras when a portion of the document is outside the field of view of the one or more cameras.

The wearable neck device may have a mechanical rotating device connected with the one or more cameras. The processor controls the mechanical rotating device to rotate the one or more cameras in order to adjust the field of view of the one or more cameras to cover the document.

The wearable neck device may have a first vibratory motor coupled to the left portion and a second vibratory motor coupled to the right portion. The first vibratory motor and the second vibratory motor provide stereo vibration data based on the determined output data.

The wearable neck device may include an antenna for establishing a wireless connection with another portable electronic device or computer having a second processor. The processors operate in conjunction with one another to analyze the image data based on the optical character recognition processing data.

The wearable neck device may also include a microphone configured to detect a speech of the user. The processor detects an input from the user requesting scanning of the environment or requesting information about the document. The processor detects the document in response to the detected input from the user.

Also described is a method for providing optical character recognition information to a user of a wearable neck device. A memory is provided for storing optical character recognition processing data corresponding to an algorithm or a set of instructions for identifying characters of documents. A document is detected by analyzing image data detected by the one or more cameras. The field of view of the one or more cameras is adjusted to be focused downwards such that the detected document is within the adjusted field of view. Image data within the adjusted field of view of the one or more cameras is captured. The image data within the adjusted field of view is analyzed using the processor and the optical character recognition processing data. Output data is determined based on the analyzed image data. A speaker outputs audio information to the user based on the output data.

Another aspect of the invention relates to a neck worn device for a user with visual impairment. The device includes a tube having a left end, a right end and a center portion positioned between the left end and the right end. A left side camera is mounted proximal to the left end for recording images. A right side camera is mounted proximal to the right end for recording images. The left side camera and the right side camera form a pair of stereo cameras. A memory is positioned within the tube for storing an optical character recognition software program. A processor is positioned within the tube and coupled to the left side camera, the right side camera and the memory. The processor identifies characters on a document using the optical character recognition software program. The processor generates a feedback signal based on the identified characters. The neck worn device has a speaker that provides audio information based on the feedback signal.

DETAILED DESCRIPTION

Apparatus, systems and methods that implement the implementations of the various features of the present application will now be described with reference to the drawings. The drawings and the associated descriptions are provided to illustrate some implementations of the present application and not to limit the scope of the present application. Throughout the drawings, reference numbers are re-used to indicate correspondence between referenced elements.

In one implementation, a smart necklace100includes an onboard processing array110, which communicates with a sensor array120, an interface array130, and a component array140. The onboard processing array110, the sensor array120, the interface array130, and the component array140are exemplary groupings to visually organize the components of the smart necklace100in the block diagram ofFIG. 1and are not limiting or necessarily representative of any physical groupings. In addition, certain implementations may have more or less components shown inFIG. 1.

The onboard processing array110includes a processor111and a memory112. The processor111may be a computer processor such as an ARM processor, DSP processor, distributed processor, or other form of central processing. The processor111may be positioned on the smart necklace100, may be a remote processor or it may be a pairing of a local and a remote processor.

The memory112may be one or any combination of the following: a RAM or other volatile or nonvolatile memory, a non-transitory memory or a data storage device, such as a hard disk drive, a solid state disk drive, a hybrid disk drive, or other appropriate data storage, and may further store machine-readable instructions, which may be loaded into the memory112and executed by the processor111. As with the processor111, the memory112may be positioned on the smart necklace100, may be positioned remote from the smart necklace100or it may be a combination of a local or remote memory.

The sensor array120includes stereo cameras121, a camera122, an inertial measurement unit (IMU)123, a global positioning system (GPS)124, and a sensor125. The stereo cameras121may be a stereo camera pair comprising two cameras offset by a stereo distance. The stereo distance may be optimized for the two cameras. The smart necklace100may have more than one pair of stereo cameras121, as will be further described below. The camera122may be a camera or other optical sensor not part of a stereo camera pair. The IMU123may be an IMU which may further comprise one or more of an accelerometer, a gyroscope, and/or a magnetometer. The GPS124may be one or more GPS units. The sensor125may be one or more sensors which provide further information about the environment in conjunction with the rest of the sensor array120. The sensor125may be, for example, one or more of a temperature sensor, an air pressure sensor, a moisture or humidity sensor, a gas detector or other chemical sensor, a sound sensor, a pH sensor, a smoke detector, a metal detector, an actinometer, an altimeter, a depth gauge, a compass, a radiation sensor, a motion detector, or other sensor.

The interface array130includes a microphone131, a speaker132, a vibration unit133, an input device134, and a display135. The microphone131may be a microphone or other device capable of receiving sounds, such as voice activation/commands or other voice actions from the user, and may be integrated with or external to the smart necklace100. The speaker132may be one or more speakers or other devices capable of producing sounds and/or vibrations. The vibration unit133may be a vibration motor or actuator capable of providing haptic and tactile output. In certain implementations, the vibration unit133may also be capable of producing sounds, such that the speaker132and the vibration unit133may be the same or integrated. The vibration unit133may include a left vibration motor in the left portion, and a right vibration motor in the right portion. This advantageously allows various combinations of haptic feedback using a left-side vibration that may differ from a right-side vibration.

The input device134may be an input device such as a touch sensor and/or one or more buttons. For example, the input device134may be a touch sensor used as a slider to adjust settings as well as act as a button for making selections, similar to a touchpad. The display135may be a display, integrated into the smart necklace100or wirelessly connected to the smart necklace100, and may be capable of displaying visual data from the stereo cameras121and/or the camera122. In other implementations, the display135may be another visual alert device, such as one or more LEDs or similar light source.

The component array140includes a battery141, an antenna142, and an input/output (I/O) port143. The battery141may be a battery or other power supply capable of powering the smart necklace100. The battery141may have a connection port for recharging, or may be wirelessly recharged, such as through induction charging. The antenna142may be one or more antennas capable of transmitting and receiving wireless communications. For example, the antenna142may be a Bluetooth or WiFi antenna, may be a radio frequency identification (RFID) antenna or reader, and/or a near field communication (NFC) unit. The I/O port143may be one or more ports for connecting additional peripherals. For example, the I/O port143may be a headphone jack, or may be a data port.

The antenna142and/or the I/O port143allows the smart necklace100to connect to another device or network for data downloads, such as updates or map information or other relevant information for a particular application, and data uploads, such as status updates. Further, the antenna142and/or the I/O port143allows the smart necklace100to communicate with other smart devices for distributed computing or sharing resources.

The smart necklace100described herein is generally a stand-alone device. However, in other implementations, the smart necklace100may be configured or optimized to work in conjunction with other devices. For example, smartphones, tablets, or other mobile devices may wirelessly connect to the smart necklace100for shared resources and processing. The mobile device may act as a display unit for the smart necklace100. The smart necklace100may further have specific protocols for interacting with mobile devices or other smart necklaces100. Additionally, the smart necklace100may connect over the internet to remote processing and/or remote storage, such as a cloud.

The smart necklace100is a lightweight, wearable smart device that is worn around the user's neck for environmental awareness, navigation, social interactions, and obstacle avoidance through real-time feedback. The smart necklace100is capable of recognizing objects around the user, in order to alert the user. For example, the smart necklace100may be used by a blind person to aid in environmental awareness and navigate safely around obstacles. The smart necklace100provides the user audio and haptic feedback through the speaker132and the vibration unit133based upon camera input from the stereo cameras121and the camera122.

In one implementation, the smart necklace100includes two pairs of stereo cameras121, which may be positioned on either side of the user's neck. Stereo cameras provide depth information in both indoor and outdoor environments. The stereo cameras121may face forward, in front of a user, to establish a field of view. The stereo cameras121may have, for example, a field of view of around 90 degrees. The stereo cameras121provide 3D information such as depth in front of the user. Stereo cameras121having different focal lengths may be provided. In various embodiments, one set of stereo cameras121may be positioned a relatively small distance apart (such as on a single side of the smart necklace100) and another pair of stereo cameras121may be positioned a relatively large distance apart (such as positioning one of the pair of stereo cameras121on a first side of the smart necklace100and the other pair of stereo cameras121on a second side of the smart necklace100). The pair of stereo cameras121having the relatively small distance apart can have a relatively short focal length, and thus provide detailed depth information for objects that are relatively close to the user. The pair of stereo cameras121having the relatively large distance apart can have a relatively long focal length, and thus provide detailed depth information for objects that are relatively far away from the user.

Additional cameras122, which may be placed to the sides of the stereo cameras121, may be wide angle cameras. Wide angle cameras can increase the field of view of the smart necklace100so that the smart necklace may have a field of view that is near 120 degrees. In various embodiments, the cameras122may be placed where needed, such as behind the user's neck to provide data for an area behind the user.

Although the cameras122may be monocular, they can provide simple recognition, even without depth or distance information. For example, the cameras122can detect moving objects in the user's periphery. The stereo cameras121and the cameras122may continuously passively recognize objects in the environment. Working in conjunction with the other sensors in the sensor array120, the smart necklace100can provide the user with guidance and navigation commands by way of audio and haptic feedback. In a preferred embodiment, the stereo cameras121are utilized in part because they can advantageously provide depth information. In another embodiment, one or more omnidirectional cameras may be utilized in addition to or in lieu of the stereo cameras12and the cameras122.

The GPS124provides location information, which works with the inertial guidance information, including velocity and orientation information, provided by the IMU123to help direct the user. The memory112may store, for example, map information or data to help locate and provide navigation commands to the user. The map data may be preloaded, downloaded wirelessly through the antenna142, or may be visually determined, such as by capturing a building map posted near a building's entrance, or built from previous encounters and recordings. The map data may be abstract, such as a network diagram with edges, or a series of coordinates with features. The map data may contain points of interest to the user, and as the user walks, the stereo cameras121and/or cameras122may passively recognize additional points of interest and update the map data.

For example, the user may give a voice command, “Take me to building X in Y campus.” The smart necklace100may then download a relevant map if not already stored, or may navigate based on perceived images from the stereo cameras121and the cameras122. As the user follows the navigation commands from the smart necklace100, the user may walk by a coffee shop in the morning, and the smart necklace100would recognize the coffee shop and the time of day, along with the user's habits, and appropriately alert the user. The smart necklace100may verbally alert the user through the speaker132. The user may use the input device134to adjust settings, which for example may control the types of alerts, what details to announce, and other parameters which may relate to object recognition or alert settings. The user may turn on or off certain features as needed.

When navigating indoors, the standalone GPS units may not provide enough information to a blind user to navigate around obstacles and reach desired locations or features. The smart necklace100may recognize, for instance, stairs, exits, and restrooms and appropriately store them in the memory112. In another example, the smart necklace100may determine empty seats for the user to navigate to, or may remember the user's specific seat in order to navigate away and subsequently return to the same seat. Other points of interest may be potential hazards, descriptions of surrounding structures, alternate routes, and other locations. Additional data and points of interest can be downloaded and/or uploaded to mobile devices and other devices, social networks, or the cloud, through Bluetooth or other wireless networks. With wireless connectivity, local processing can be reduced, as high level data and processing may be available from the cloud or other remote data centers.

The smart necklace100may determine paths for navigation, which may be further modified for the user's needs. For example, a blind person may prefer routes that follow walls. Using the IMU123and/or the GPS124and other sensors, the smart necklace100can determine the user's location and orientation to guide them along the path, avoiding obstacles. The vibration unit133and the speaker132provide audio and haptic cues to help guide the user along the path.

For example, the speaker132may play a command to move forward a specified distance. Then, special audio tones or audio patterns can play when the user is at a waypoint, and guide the user to make a turn through additional tones or audio patterns. A first tone, audio pattern or vibration can alert the user to the start of a turn, such as a single tone or a vibration from the left side of the smart necklace may indicate a left turn. A second tone, audio pattern or vibration can alert the user that the turn is complete such as two tones, or the vibration may stop, such as the left side ceases to vibrate when the turn is complete. Different tones, patterns or vibrations may also signify different degrees of turns, such as a specific tone for a 45 degree turn and a specific tone for a 90 degree turn. Alternatively or in addition to tones and vibrations, the smart necklace100may provide verbal cues, similar to a car GPS navigation command.

High level alerts may also be provided through audio feedback. For example, as the smart necklace100reaches a predetermined distance—such as a foot or other value which may be stored in the memory112and may be adjusted—from an obstacle or hazard, the speaker132and/or the vibration unit133may provide audible alerts. As the smart necklace100gets closer to the obstacle, the audible alerts may increase in intensity or frequency.

The vibration unit133may include a left vibration motor in the left portion of the smart necklace100and a right vibration motor in the right portion of the smart necklace100for providing stereo haptic feedback to the user. Vibration patterns on the left portion can be different than vibration patterns on the right portion. In this manner, different combination of left/right vibration patterns can convey more variety of useful information to the user (as opposed to outputting the same pattern in both left and right vibration). For example, certain vibration patterns on the left that are lacking on the right may be used to signal to the user that the user should turn left.

The microphone131may detect additional environmental data, such as sounds of moving cars or other possible hazards. The microphone131may work in conjunction with the speaker132, and may be placed away from the speaker132to prevent interference. The microphone131may alternatively work in conjunction with an attached audio device, such as bone conduction devices, to provide the user with audio feedback without broadcasting the audio feedback.

The smart necklace100may improve social interactions. For example, the smart necklace100may recognize faces in a room to identify potential friends, and provide the user with audio feedback identifying friends. The stereo cameras121and/or the camera122may be further able to determine additional details about persons, such as moods or expressions, or if they are engaging in physical activities, in order to alert the user. For example, the potential friend may extend a hand for a handshake or a “high five,” and the smart necklace100may use audio or haptic feedback to notify the user. The microphone131may recognize voices of other persons to identify and appropriately notify the user, or may recognize a new voice to save for future identification.

Although the smart necklace100is described with respect to a blind user, the smart necklace100may be used in other applications. For example, the smart necklace100may be used by peace officers and law enforcement officers as a recorder which provides additional environmental awareness. The smart necklace100may be further used by athletes to record sports in a real-time, first person view. For example, performing certain actions such as a swing can be recorded, including inertial motions, to analyze the motions. The smart necklace100may also be used in hazardous environments to provide additional safety warnings. For example, the smart necklace100may be used in a factory to provide a factory worker additional warning about possible hazardous conditions or obstacles.

In such applications, the sensor125may be specifically chosen to provide particularly relevant measurements. For instance, in an environment with harmful gas, the sensor125may detect dangerous levels of gas and accordingly alert the user. The sensor125may provide low-light viewing, or the stereo cameras121and/or the camera122may be capable of night vision, to provide the user with additional environmental awareness in low-light conditions, such as outdoors at night or photo-sensitive environments. The sensor125, the stereo cameras121and/or the camera122may be adapted to detect a light spectrum other than the visible light spectrum. The smart necklace100can be a memory device to aid persons, such as Alzheimer's patients. The smart necklace100can aid in shopping or otherwise navigating inventories by helping to keep track of goods. The antenna142may be an RFID or NFC reader capable of identifying RFID or NFC tags on goods.

In certain embodiments, the smart necklace100is designed to accommodate blind or partially blind users. In such embodiments, a low-light viewing or night-vision camera (e.g., infrared camera) may also be utilized. For example, a camera122may be directed to normal lighting and another directed to night vision. For example, a blind user may be more likely to turn off the lights because he/she does not depend on the lighting. The smart necklace100would still function properly by processing images of the night-vision camera. The image processed may be limited in night-vision. For example, facial recognition may not be feasible, but the presence of another person can be detected. As a result, helpful information can be provided to the user.

The smart necklace100may include an infrared camera in combination with the stereo cameras121and/or camera122. For example, a wide angle camera122and/or stereo cameras121may be utilized for image detection for normal lighting situations and an infrared camera may be utilized for image detection for darker situations.

Because the smart necklace100may be used for environmental awareness and detection, other light spectrums may be useful. The visible light spectrum allows humans to detect certain details that other light spectrums may not provide. However, other light spectrums may provide certain details that human visible light spectrum cannot provide. Details of certain objects may not be easily detected by the visible light spectrum during a cloudy or foggy day. Another spectrum of light may provide better details of objects during certain conditions. These spectrums may include extreme ultraviolet, near infrared, mid infrared, far infrared, etc. For maximum efficiency and object detection, different sensors125, stereo cameras121and/or cameras122may be provided for detecting various light spectrum data. In some embodiments, a single camera122is provided that detects a spectrum of light other than the visible light spectrum.

Referring now toFIG. 2, a flowchart of a method200of adjusting object detection parameters, object recognition parameters, or both object detection parameters and object recognition parameters is schematically depicted. In some implementations, the method200may be implemented as logic within the machine readable instructions that, when executed by the processor111, automatically adjust object detection parameters, object recognition parameters, or both object detection parameters and object recognition parameters. It is noted that, while the method200depicts a specific sequence, additional implementations of the present disclosure are not limited to any particular sequence.

Referring now toFIG. 2, at block210the smart necklace100receives image data representative of the environment. As noted above, in some implementations, the smart necklace100is configured to acquire video or image data, which may be video frames, of the field of view of the user from the stereo cameras121and the cameras122, and to then send the acquired image data of the environment to the processor111and/or the memory112for storage and/or processing. In some implementations, the smart necklace100may receive image data from a source external to the smart necklace100, such as via the antenna142through a wireless network.

The image data received at block210may be data of a variety of forms, such as, but not limited to red-green-blue (“RGB”) data, depth image data, three dimensional (“3D”) point data, and the like. In some implementations, the smart necklace100may receive depth image data from an infrared sensor or other depth sensor, such as an infrared sensor or depth sensor integrated with the stereo cameras121and/or the camera122. In other implementations that include a depth sensor (e.g., an infrared sensor), the depth sensor may be separate from the stereo cameras121and/or the camera122.

Still referring toFIG. 2, at block220, the machine readable instructions stored in the memory112, when executed by the processor111, cause the smart necklace100to detect a candidate object, with the onboard processing array110, based on the image data received at block210. In some implementations, the onboard processing array110may detect the candidate object by identifying a candidate region of the received image data, such as a region of the image that includes high entropy. For example, the onboard processing array110may detect a high entropy region in the acquired target image data that includes a spray bottle. In some implementations, the onboard processing array110may utilize a sliding window algorithm to identify the candidate region of the received image data. In implementations, the onboard processing array110may detect the candidate object by utilizing a feature descriptor algorithm or an image descriptor algorithm, such as scale-invariant feature transform (“SIFT”), speeded up robust feature (“SURF”), histogram of oriented gradients (“HOG”), generalized search tree (“GIST”), fast retina keypoint (“FREAK”), and binary robust invariant scalable keypoints (“BRISK”), and the like. In some implementations, the onboard processing array110may bias detections to one or more spatially located regions of interest based on application, scene geometry and/or prior information.

The onboard processing array110includes at least one object detection parameter to facilitate the detection of the candidate object. In some implementations, the at least one object detection parameter is a window size, a noise filtering parameter, an estimated amount of light, an estimated noise level, a feature descriptor parameter, an image descriptor parameter, or the like.

Still referring toFIG. 2, at block230, the machine readable instructions stored in the memory112, when executed by the processor111, cause the smart necklace100to recognize an object, with the onboard processing array110, based on the image data received at block210. In some implementations, the object recognition module may recognize the object based on a candidate region identified by the onboard processing array110.

In some implementations, the onboard processing array110may recognize the candidate object by utilizing a feature descriptor algorithm or an image descriptor algorithm, such as scale invariant feature transform (“SIFT”), speeded up robust feature (“SURF”), histogram of oriented gradients (“HOG”), generalized search tree (“GIST”), fast retina keypoint (“FREAK”), and binary robust invariant scalable keypoints (“BRISK”), and the like. In some implementations in which the onboard processing array110utilizes a feature descriptor or image descriptor algorithm, the onboard processing array110may extract a set of features from a candidate region identified by the onboard processing array110. The onboard processing array110may then access a reference set of features of an object recognition reference model from an object recognition database stored in the memory112and then compare the extracted set of features with the reference set of features of the object recognition reference model. For example, the onboard processing array110may extract a set of features from the high entropy region of the acquired target image data that includes a bottle and compare the extracted set of features to reference sets of features for one or more reference bottle models. When the extracted set of features match the reference set of features, the onboard processing array110may recognize an object (e.g., recognizing a bottle when the extracted set of features from the high entropy region of the acquired target image data that includes the bottle match the reference set of features for a reference bottle model). When the extracted set of features does not match the reference set of features, an object recognition error has occurred (e.g., an object recognition error indicating that no object recognition reference model matches the candidate object). When an object recognition error has occurred (e.g., referring to the example, no reference bottle model exists in the memory112), the at least one object detection parameter may be adjusted to improve the accuracy of the object detection module, as described below with reference to block225.

In some implementations, the object recognition module may assign an identifier to the recognized object. For example, the identifier may be an object category identifier (e.g., “bottle” when the extracted set of features match the reference set of features for the “bottle category” or “cup” when the extracted set of features match the reference set of features for the “cup” object category) or a specific object instance identifier (e.g., “my bottle” when the extracted set of features match the reference set of features for the specific “my bottle” object instance or “my cup” when the extracted set of features match the reference set of features for the specific “my cup” object instance).

The onboard processing array110includes at least one object recognition parameter to facilitate the recognition of the object. In some implementation, the at least one object recognition parameter is a window size, a noise filtering parameter, an estimated amount of light, an estimated noise level, a feature descriptor parameter, an image descriptor parameter, or the like.

Still referring toFIG. 2, at block240, the machine readable instructions stored in the memory112, when executed by the processor111, cause the smart necklace100to send control signals to the vibration unit133and/or the speaker132to provide appropriate haptic and audio feedback to the user. For example, if the object recognized is categorized as an obstacle, the vibration unit133may vibrate at an increasing rate as the smart necklace approaches it. If the object is categorized as a hazard, the speaker132may play a warning sound. If the object is categorized as a point of interest, the speaker132may play an appropriate notice, or may remain silent. As noted above, when an object recognition error has occurred, the at least one object detection parameter may be adjusted to improve the accuracy of the onboard processing array110.

Still referring toFIG. 2, at block225, the machine readable instructions stored in the memory112, when executed by the processor111, cause the smart necklace100to adjust at least one object detection parameter of the onboard processing array110, with a detection tuner module, when an object recognition error has occurred. The detection tuner module may be implemented as instructions executed by the processor111and data stored on the memory112. By way of non-limiting example, in some implementations, the detection tuner module may adjust the window size utilized by the onboard processing array110when an object recognition error has occurred. In some implementations, the detection tuner module includes a detection tuner model and the detection tuner model adjusts the at least one object detection parameter based on the object recognition error. In some implementations, the detection tuner model maps the object recognition error to the adjusted at least one object detection parameter. In some implementations, the detection tuner model is a learned correlation model, such as a support vector machine (“SVM”) model.

Still referring toFIG. 2, at block235, the machine readable instructions stored in the memory112, when executed by the processor111, cause the smart necklace100to adjust at least one object recognition parameter of the onboard processing array110, with a recognition tuner module, when object recognition error has occurred. The recognition tuner module may be implemented as instructions executed by the processor111and data stored on the memory112. By way of non-limiting example, in some implementations, the recognition tuner module may adjust the window size utilized by the onboard processing array110when object recognition error has occurred. In some implementations, the recognition tuner module includes a recognition tuner model and the recognition tuner model adjusts the at least one object recognition parameter based on the object recognition error. In some implementations, the recognition tuner model maps the object recognition error to the adjusted at least one object recognition parameter. In some implementations, the recognition tuner model is a learned correlation model, such as a support vector machine (“SVM”) model.

FIGS. 3A-3Cpresent one example of a method of object recognition according to an implementation of the present disclosure. InFIG. 3A, for example, the first visual data306corresponds to a 2-D image of the target object310positioned on a plane324(e.g., a table). The second visual data308corresponds to 3-D depth data of the target object310positioned on the plane324. Category object recognition is performed for analyzing, using the processor111and the first visual data306. The first visual data306is analyzed based on a plurality of object models stored in a database, which may be stored in the memory112. For example, the plurality of object models may include primitive shapes such as cylinders, boxes, and the like associated with corresponding parameters defining the primitive shapes. For example, the processor111may determine whether any portion of the first visual data306corresponds in shape, structure, or in terms of other pose information, to the plurality of object models stored in the database. Each of the plurality of object models may have at least one parameter. For example, an object model may be a cylinder with parameters of a height and a radius. For example, an object model may be a box with three parameters of a width, a height, and a length.

When the processor111searches for an object model of the plurality of object models, more than one object model may be similar in shape or structure to a portion of the first visual data306. For example, a body of a bottle (e.g., the target object310) may be similar in shape or structure to either a cylinder or a box. The processor111is configured to determine which of the plurality of object models has the closest fit for the analyzed portion of the first visual data306. For example, the processor111may assign a score (for example, a recognition accuracy percentage) as to the degree of similarity between a particular object model of the plurality of object models and the analyzed portion of the first visual data306. For example, the processor111may choose the object model of the plurality of object models associated with the highest associated score (e.g., recognition accuracy percentage), as the object model that corresponds to the analyzed portion of the first visual data306. As such, in one implementation, the processor111determines the parameters of the chosen object model.

As described below, the plurality of object models are not fixed. The stored object models and their corresponding parameters may be supplemented or modified. In addition or in the alternative, new category object models may be learned and stored in the database based on the recognized target objects. The discussion at this juncture assumes that the method is detecting the target object310for the first time, and objects having similar shapes, structure, or pose information to the target object310as a whole are not yet encountered and stored.

Referring toFIG. 3B, an example of the category object recognition of the method is illustrated. For example, the processor111may examine the first visual data306adjacent to, around, or within the sliding enclosure312from left to right, starting from the top left corner of the 2-D image represented by the first visual data306moving right thereafter in the direction314. The processor111may recognize objects within the first visual data306that are similar in shape or structure to an object model of the plurality of object models stored in the database. In other implementations, instead of the sliding enclosure312, the visual data set304, the first visual data306, the second visual data308, or combinations thereof may be examined as a whole to determine whether any portion of the first visual data306matches an object model stored in the database.

FIG. 3Cillustrates a result of the category object recognition. The processor111may recognize that the target object310is similar to one of the object models. The first enclosure350may be a bounding box, a bounding circle, or any other shape without limiting the scope of the invention. The first enclosure350has a first center point316. When the first enclosure350is a bounding box, the first center point316is the point with approximately equal distance from each side of the bounding box. When the first enclosure350is a bounding circle, the first center point316may be the center of the bounding circle. In one embodiment, the processor111may determine the first center point316such that the first center point316is positioned on, corresponds to, or falls within a portion of the visual data set304corresponding to the target object310. The target object310may, for example, be positioned within, around, or adjacent to the first enclosure350. The processor111determines that a first target data (which is a portion of the first visual data306) corresponds to the target object310to recognize the target object310.

Although the method described above uses a bottle as an exemplary object, the method may be used to recognize points of interest and other features, such as stairs, empty seats or buildings.

Referring now toFIG. 4, which shows a flowchart400of a method of estimating position or orientation based on slice descriptors. The onboard processing array110receives omni-directional image data representative of a panoramic field of view from the stereo cameras121and the cameras122. In some implementations, the stereo cameras121and the cameras122operate within a physical environment and is configured to acquire omni-directional image data, and to then send the acquired omni-directional image data of the physical environment to the onboard processing array110for storage and/or processing. In some implementations, the onboard processing array110may receive omnidirectional image data from a source external to the smart necklace100, such as via the antenna142. The acquired omni-directional image data may be in the form of digital video and/or one or more digital photographs.

The onboard processing array110segments the omni-directional image data into a plurality of image slices. In one exemplary implementation, the received omni-directional image is segmented into eight slices (S1, S2, S3, S4, S5, S6, S7, and S8). In some implementations, the omni-direction image may be segmented into any number of slices. In some implementations, the number of slices may be between 8 and 36. However, it should be understood that the number of slices may be less than 8 or greater than 36.

Each of the plurality of slices is representative of at least a portion of the panoramic field of view of the omni-directional image data or the partially panoramic field of view of the omni-directional image data. In some implementations, the plurality of image slices includes a middle image slice (e.g., slice S2), a preceding image slice (e.g., slice S1), and a subsequent image slice (e.g., slice S3), such that a field of view of the middle image slice (e.g., slice S2) is adjacent to or overlaps a preceding field of view of the preceding image slice (e.g., slice S1) and the middle field of view of the middle image slice (e.g., slice S2) is adjacent to or overlaps a subsequent view of the subsequent image slice (e.g., slice S3).

In some implementations, each image slice of the plurality of image slices is representative of an equal portion of the panoramic field of view of the omni-directional image data and the collective fields of view of the plurality of image slices is the same as the panoramic field of view of the omni-directional image data. For example, each of the eight slices captures an eighth of the full panoramic view of the omnidirectional image data and the collective field of view of the eight image slices is the same as the panoramic field of view of the omni-directional image data. In some implementations, the field of view of a first slice of the plurality of views may be greater than a field of view of a second slice of the plurality of slices. In some implementations, the collective fields of view of the plurality of slices may be smaller than the full panoramic field of view. In some implementations, the field of views of neighboring slices may overlap.

The onboard processing array110calculates a slice descriptor for each image slice of the plurality of image slices. As used herein, “slice descriptor” refers to a description of the visual features (e.g., color, texture, shape, motion, etc.) of the image data of a particular slice of the omni-directional image data. For example, a slice descriptor d1is calculated for slice S1, a slice descriptor d2is calculated for slice S2, a slice descriptor d3is calculated for slice S3, a slice descriptor d4is calculated for slice S4, a slice descriptor d5is calculated for slice S5, a slice descriptor d6is calculated for slice S6, a slice descriptor d7is calculated for slice S7, and a slice descriptor d8is calculated for slice S8.

In some implementations, the slice descriptor may be calculated using an algorithm, such as scale-invariant feature transform (“SIFT”), speeded up robust feature (“SURF”), histogram of oriented gradients (“HOG”), generalized search tree (“GIST”), fast retina keypoint (“FREAK”), and binary robust invariant scalable keypoints (“BRISK”), and the like. However, it should be understood that other algorithms may be used to calculate the slice descriptor. In some implementations, the slice descriptor may include a decimal vector. In some implementations, the slice descriptor may include a binary vector. In other implementations, the slice descriptor may be represented in a format other a binary vector or a decimal vector. Depth information resulting from the application of stereo algorithms may also be used to calculate the slice descriptor.

The onboard processing array110generates a current sequence of slice descriptors for the omni-directional image data received. The current sequence of slice descriptors includes the calculated slice descriptor for each image slice of the plurality of image slices. For example, node n1includes the slice descriptor d1corresponding to slice S1, node n2includes the slice descriptor d2corresponding to slice S2, node n3includes the slice descriptor d3corresponding to slice S3, node n8includes the slice descriptor d8corresponding to slice S8, etc.

In some implementations, the current sequence of slice descriptors may be structured such that a middle node (e.g., node n2) corresponds to a middle image slice (e.g., slice S2), a preceding node (e.g., node n1) corresponds to a preceding image slice (e.g., slice S1), and a subsequent node (e.g., node n3) corresponds to a subsequent image slice (e.g., slice S3). The preceding node (e.g., node n1) is linked to the middle node (e.g., node n2), and the middle node (e.g., node n2) is linked to the subsequent node (e.g., node n3).

In some implementations, the current sequence of slice descriptors are stored in the memory112. In some implementations, the memory112may include a database of reference sequences of slice descriptors, each of which corresponds to a previously processed omni-directional image encountered by the onboard processing array110.

In some implementations, the current sequence of slice descriptors may be stored in the memory112as a current linked list of slice descriptors. In implementations in which the current sequence of slice descriptors is stored in the memory112as a current linked list of slice descriptors, each node of the linked list may be linked to the subsequent node of the linked list (e.g., node n1is linked to node n2, node n2is linked to node n3, etc.). In some implementations, the current sequence of slice descriptors may be stored in the memory112as a circular linked list of slice descriptors, such that the first node is linked to the second node (e.g., node n1is linked to node n2), the second node is linked to the third node (e.g., node n2is linked to node n3), . . . , and the last node is linked back to the first node (e.g., node n8is linked to node n1). In some implementations, the current sequence of slice descriptors may be stored in the memory112as a current doubly linked list of slice descriptors. It should be understood that in other implementations, the current sequence of slice descriptors may be stored in the memory112using a data structure other than a linked list, such as an array, and the like.

While the omni-directional image received was not unwarped prior to segmenting the omni-directional image, in other implementations, the omni-directional image may be unwarped prior to segmentation.

Returning toFIG. 4, at block402, the machine readable instructions stored in the memory112, when executed by the processor111, may cause the onboard processing array110to access a reference sequence of slice descriptors in the memory112. For example, in the implementation depicted inFIG. 4, the reference sequence of slice descriptors includes a reference slice descriptor d3corresponding to a reference node r1, a reference slice descriptor d4corresponding to a reference node r2, a reference slice descriptor d5corresponding to a reference node r3, a reference slice descriptor d6corresponding to a reference node r4, a reference slice descriptor d7corresponding to a reference node r5, a reference slice descriptor d8corresponding to a reference node r6, a reference slice descriptor d1corresponding to a reference node r7, and a reference slice descriptor d2corresponding to a reference node r8.

Still referring toFIG. 4, at block404, the machine readable instructions stored in the memory112, when executed by the processor111, may cause the onboard processing array110to determine whether the current sequence of slice descriptors matches the reference sequence. In some implementations, whether the current sequence of slice descriptors matches the reference sequence of slice descriptors is determined by determining a current order of slice descriptors, determining a reference order of slice descriptors, and comparing the current order of slice descriptors to the reference order of slice descriptors. For example, a current order of slice descriptors in the embodiment depicted inFIG. 4may be determined as {d1, d2, d3, d4, d5, d6, d7, d8}. A reference order of slice descriptors in the embodiment depicted inFIG. 4may be determined as {d3, d4, d5, d6, d7, d8, d1, d2}. The current order of slice descriptors {d1, d2, d3, d4, d5, d6, d7, d8} may be compared to the reference order of slice descriptors {d3, d4, d5, d6, d7, d8, d1, d2} in order to determine whether the current order of slice descriptors matches the reference order of slice descriptors.

In some implementations, the current sequence of slice descriptors is a current circular linked list of slice descriptors and the reference sequence of slice descriptors is a reference circular linked list of slice descriptors. In such implementations, the current order of slice descriptors may be determined by traversing the current circular linked list of slice descriptors starting at a current starting node (e.g., the current order of slice descriptors may be determined to be {d1, d2, d3, d4, d5, d6, d7, d8} by traversing the current circular linked list starting from node n1of the current circular linked list of slice descriptors). The reference order of slice descriptors may be determined by traversing the reference circular linked list of slice descriptors starting at a reference starting node (e.g., the reference order of slice descriptors may also be determined to be {d1, d2, d3, d4, d5, d6, d7, d8} by traversing the reference circular linked list starting from node r7of the reference circular linked list of slice descriptors). The current sequence of slice descriptors matches the reference sequence of slice descriptors when the current order of slice descriptors is the same as the reference order of slice descriptors. In the embodiment depicted inFIG. 4, the current sequence of slice descriptors may be determined to match the reference sequence of slice descriptors because the reference order of slice descriptors when traversing the reference circular linked list of slice descriptors starting from node r7is the same as the current order of slice descriptors when traversing the current circular linked list of slice descriptors starting from node n1.

Still referring toFIG. 4, at block406, the machine readable instructions stored in the memory112, when executed by the processor111, may cause the onboard processing array110to estimate an orientation or position based on the current sequence of slice descriptors and the reference sequence of slice descriptors. For example, differences between the current sequence of slice descriptors and the reference sequence of slice descriptors may be used to determine a current position or orientation with reference to a known position or orientation associated with the reference sequence of slice descriptors. In some implementations, standard filtering techniques, such as the extended Kalman filter, the particle filter, and the like may be used to determine the current position or orientation based on the comparison between the current sequence of slice descriptors and the reference sequence of slice descriptors.

FIGS. 5A-9illustrate various exemplary implementations of the smart necklace100. AlthoughFIGS. 5A-9depict specific locations of components, in other implementations the exact locations and configurations may vary, as more or less components may be added or rearranged depending on specific applications. For example, the onboard processing array110may generally be located in the band portion of the smart necklace. Other hidden components, such as the IMU123, the GPS124, the sensor125, the vibration unit133, the battery141, and the antenna142, may be placed along the inside of the smart necklace as needed. The use of specialized sensors may require specific sensor placement and rearrangement of the other components. In addition, although the various implementations described generally take on a U-shape, in other implementations, other shapes, such as an O-shape or Ω-shape, may be used.

Turning toFIGS. 5A-5C,FIGS. 5A-5Cdepict one implementation of a smart necklace500having a “stalk” design. The smart necklace500has a left stalk551and a right stalk552, connected by a tube554, which may be a flexible tube for added comfort. The smart necklace500corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left stalk551, the right stalk552, and the tube554. Specifically, the tube554defines a cavity which houses the unseen components of the smart necklace500.

The left stalk551and the right stalk552each have a pair of stereo cameras521, which may be offset by a stereo distance557. Although inFIG. 5Athe stereo distance557is the same for both pairs of stereo cameras521, in other implementations the stereo distance557may be different for each pair of stereo cameras521. Cameras522are placed to the sides of the stereo cameras521to increase the field of view.

Speakers532are placed above the left stalk551and the right stalk552. A microphone531is placed away from the speakers532. A button553may be used as a power button or on/off switch, or other input. A touch sensor534, which may be delimited with bump indicators558, may provide another input for the user. A headphone jack543provides a port for attaching, for example, a 3.5 mm headphone set. A data port563may provide an additional port for data connections. The headphone jack543and the data port563may be located on the underside of the smart necklace500, as seen inFIG. 5C. A texture strip555, which may be rubber or silicone, lines the tube554to provide added cushioning and grip. In addition, the smart necklace500may have braille556for identification or a label for blind persons.

In various embodiments, the processor111, the memory112, the IMU123, the GPS124, the battery141, and all other components of the smart necklace100are distributed throughout the smart necklace100. For the comfort of the user, it is desired for the weight of the smart necklace100to be distributed evenly between the right side and the left side of the smart necklace. If the weight is significantly greater on one side of the smart necklace100than the other side, then the smart necklace500may be uncomfortable for the user to wear.

In various embodiments, all components of the smart necklace100are balanced between the front and the back of the smart necklace100. The embodiment illustrated inFIGS. 5A-5Chave a balance of weight that is almost all directed to the front of the smart necklace500(i.e., most components are positioned on the left stalk551and the right stalk552). The tube554may be soft and/or flexible for added comfort.

If too much weight is distributed towards the back (the neck area) of the smart necklace100, then the smart necklace100may be pulled backwards by the weight of the components. This can result in the smart necklace100falling off of the user or in unsteady imaging.

If the smart necklace100shifts too much when the user moves (because of the heavy weight towards the back), then the stereo cameras121and/or the camera122may shift positions by a significant amount and not be able to capture data from a uniform vantage point. By evenly distributing the weight between the front and the back, the weight towards the front of the smart necklace100will pull the front of the smart necklace100down so that the position of the stereo cameras121and the camera122does not shift while the user is moving. It has been discovered that optimum weight distribution may be an even distribution between the front and the back of the smart necklace100, or slightly towards the front of the smart necklace100.

FIGS. 6A-6Cdepict one implementation of a smart necklace600having a “pod” design, which may resemble two peapods connected by a band. The smart necklace600has a left pod651and a right pod652, connected by a band654, which may be a flexible band for added comfort. The smart necklace600corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left pod651, the right pod652, and the band654. Specifically, the band654defines a cavity which houses the unseen components of the smart necklace500.

The left pod651and the right pod652each have a pair of stereo cameras621. Cameras622are placed to the sides of the stereo cameras621to increase the field of view. Each of the stereo cameras621and cameras622may be placed on its own “pea,” as seen in the three peas inFIGS. 6A and 6C. In other implementations, more or less peas may be used to house additional cameras or sensors. In certain other implementations, one or more of the peas may be removable and modular, to create custom combinations of sensors. For example, additional peas may be screwed on, magnetically attached, or otherwise removably attached, with electrical connector points between adjacent peas for sending power and signals.

Speakers632are placed above the left pod651and the right pod652. Microphones631are placed behind the left pod651and the right pod652and away from the speakers632, as seen inFIG. 6B. A headphone jack643, which may be located on an underside of the smart necklace600, provides a port for attaching, for example, a 3.5 mm headphone set. A data port663, which may be a universal serial bus (USB) port or other similar data port, provides an additional data connection to other devices. A touch sensor634, which may be delimited with bump indicators658, may provide another input for the user. A button653may be used as a power button or on/off switch, or other input. A texture strip655, which may be rubber or silicone, lines the band654to provide added cushioning and grip.

In addition, the smart necklace600may have braille656for identification or a label for blind persons. LEDs635, placed near the bottom of the left pod651and the right pod652, provide visual indicators, such as status indicators, and may have different colors or flashing patterns to indicate various statuses. An overmold671, which may be rubber or silicone, lines each of the left pod651and the right pod652, and may be soft and flexible to provide additional grip and cushioning.

FIG. 6Dis an image of a user690utilizing the smart necklace600inFIG. 6Aalong with a smart watch691. Instead of the smart watch691, the user690may be using a smartphone, a tablet, or the like. The user690may be in a grocery store and have created an electronic grocery list that is stored in remote memory or a memory of the smart watch691. In various embodiments, the grocery list may be stored in the memory112of the smart necklace600. The smart necklace600may be in data communication with the smart watch691such that the smart necklace600can access the grocery list.

As the user690strolls through the grocery store, the stereo cameras621and/or the cameras622may scan the aisles and detect items. The smart necklace may determine whether each detected item in the store matches an item on the grocery list. When an item is detected in the grocery store, the smart necklace600may alert the user690to the presence of a matched item. The smart necklace600may inform the user690of the precise item and the precise location of the item. The smart necklace600may also or instead direct the user690to the precise item via audio tones/patterns or vibration patterns. The smart necklace600may determine directions given to the user690by comparing the location of the smart necklace600and/or an arm of the user690to the precise location of the item, and guide the user's arm to the item.

FIGS. 7A-7Bdepict one implementation of a smart necklace700having a “paddle” design, which may resemble two paddles connected by a band. The smart necklace700has a left paddle751and a right paddle752, connected by a band754, which may be a flexible band for added comfort. The smart necklace700corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left paddle751, the right paddle752, and the band754. Specifically, the band754defines a cavity which houses the unseen components of the smart necklace500.

The left paddle751and the right paddle752each have a pair of stereo cameras721. Cameras722are placed to the sides of the stereo cameras721to increase the field of view. Speakers732are placed above the left paddle751and the right paddle752. Microphones731are placed on the sides of the left paddle751and the right paddle752and towards the inside of the U-shape of the smart necklace700, as seen inFIG. 7A. A headphone jack743provides a port for attaching, for example, a 3.5 mm headphone set. A data port763may provide an additional port for data connections. A touch sensor734, which may be delimited with bump indicators758, may provide another input for the user. A button753may be used as a power button or on/off switch, or other input.

An overmold755, which may be rubber or silicone, may line a portion or a majority of the smart necklace700to provide added cushioning and grip. In addition, the smart necklace700may have braille756for identification or a label for blind persons. LEDs735, placed near the bottom of the left paddle751and the right paddle752, provide visual indicators, such as status indicators, and may have different colors or flashing patterns to indicate various statuses.

FIGS. 7C-7Dillustrate a smart necklace701, which is similar to the smart necklace700with a modified paddle design. A modified paddle780, which may be either a right paddle or a left paddle or both, has the overmold755, but narrows at a neck portion785. The modified paddle780is configured to have removable pucks, such as a left puck781and a right puck782inFIG. 7D. The overmold755has a large, oblong puck opening783for housing the left puck781or the right puck782. The overmold755further includes a side opening784to allow the cameras722to have a side view. The left puck781and the right puck782may be removed from the overmold755through the puck opening783.

In other implementations, the left puck781and the right puck782may be slid out of an additional hole in the overmold755, or other similar method as needed. For example, in other implementations the left puck781and the right puck782may be magnetically attachable, and may further be connected by a retractable tether. The neck portion785may contain a connection port for the left puck781and the right puck782to send signals to and from the smart necklace700, and to receive power. The left puck781and the right puck782may further include its own battery, respectively.

InFIG. 7D, the left puck781has stereo cameras721separated by a stereo distance757. The right puck782has the same stereo distance757. However, in other implementations the stereo distance757may be different. In yet other implementations, the stereo distance757may be variable, giving the user the option to remove the left puck781and focus on a smaller object, or to get a wider view. For example, the user may wish to have more information about a specific item. By placing the left puck781near the item, the stereo cameras721may be better able to detect finer details and provide isolated object recognition to provide the user more descriptive information. In certain other implementations, the left puck781and/or the right puck782may have other sensors, such that the pucks are interchangeable with cameras or sensors as needed in a particular application.

Turning now toFIG. 8,FIG. 8depicts one implementation of a smart necklace800having a “twist” design, which may resemble a twisted band or tube. The smart necklace800has a left prong851and a right prong852, connected by a band854, which may be a flexible band for added comfort. The smart necklace800corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left prong851, the right prong852, and the band854. The left prong851and the right prong852each have a pair of stereo cameras821. Cameras822are placed to the sides of the stereo cameras821to increase the field of view. An overmold855, which may be rubber or silicone, may line a portion or a majority of the smart necklace800to provide added cushioning and grip. As seen inFIG. 8, the overmold855twists around the smart necklace800, starting from the left prong851and twisting around the band854. An LED835, placed near the bottom of at least the left prong851, provides a visual indicator, such as a status indicator, and may have different colors or flashing patterns to indicate various statuses.

Turning now toFIG. 9A,FIG. 9Adepicts one implementation of a smart necklace900ahaving a removable tube981awith a camera, according to an implementation of the present disclosure. The smart necklace900acorresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left portion951a, the right portion952a, and the band954a. The left portion951aand the right portion952amay each have a pair of stereo cameras (not shown) similarly as discussed above with respect toFIGS. 5A-8. In one embodiment, the smart necklace900amay include input devices (e.g., a microphone), output devices (e.g., a speaker and/or haptic feedback), other units, or combinations thereof as described above with respect to embodiments shown inFIGS. 5A-8.

Similar to the embodiments described above with respect toFIGS. 7A-7D, a removable tube981amay be equipped with a camera. For example, the removable tube981may be magnetically attachable; attachable with a clip; connected by a retractable tether, wire or cable; or combinations thereof. For example, the user may wish to have more information about a specific item. By placing the camera of the removable tube981anear the item, the smart necklace900may be better able to detect finer details and provide isolated object recognition to provide the user more descriptive information. In other embodiments, removable tubes981amay be provided on the right portion952aor both portions951aand952a, or other portions of the smart necklace900. In addition to or instead of the removable tube981, other removable cameras may be provided using various other attaching mechanisms based on practical design and aesthetic concerns in order to allow the user to enhance detection of information.

The modularity of the smart necklace900provides for customization of the smart necklace900. In some embodiments, it may be preferred for the smart necklace900to include a faster processor111, a higher-quality camera922, a night vision sensor125, etc. instead of standard components. Because the removable tube981acan be easily detached and attached, a new removable tube981acan be provided having the desired components. This provides for the smart necklace900to be used in various situations instead of having to purchase smart necklaces900having different components thereon.

FIG. 9Bis a view of a smart necklace with a removable tube981bconnected to the smart necklace900bvia a retractable cord909, according to an alternative implementation. Smart necklace900bcorresponds to the smart necklace100, and has similar components as the smart necklace900a, which are located within the left portion951b, the right portion952b, and the band954b. A feature of this embodiment is that the removable tube981bis configured to be removed from an end of the left portion951busing a retractable cord909b. The retractable cord909bis preferably long enough for the user to remove the removable tube981band direct a camera positioned on the removable tube981btowards any object.

A benefit of the smart necklace900bis that the removable tube981bcannot become lost. If a user removes the removable tube981bin order to enhance detection of information and the user forgets to reattach the removable tube981b, the removable tube981bwill still be attached to the smart necklace900b.

FIG. 10Ais a view of a smart necklace1000with indicators1035, according to an implementation of the present disclosure. The smart necklace1000corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left portion1051, the right portion1052, and the band1054. The smart necklace1000includes a stereo cameras pair1021distributed between the left portion1051and the right portion1052. The left portion1051also includes a button1034and three indicators1035. The button1034and/or one or more indicators1035may be positioned on the right portion1052instead of the left portion1051.

The button1034may act as an input to the smart necklace1000. The button1034may function as a power button for turning the smart necklace1000on and off. A press of the button1034may also serve other purposes, such as signaling to the smart necklace1000that a verbal instruction will be provided from the user, changing modes of operation of the smart necklace1000, etc.

The indicators1035may be LEDs that are capable of displaying one or more colors. The indicators1035can provide information about the status of the smart necklace. One indicator1035amay indicate a power status of the smart necklace1000. Another indicator1035bmay indicate a mode of operation of the smart necklace1000. Another indicator1035cmay indicate connectivity to a remote device, such as a smartphone, a smart watch, a tablet, etc. Any of the indicators1035may indicate the information mentioned above or any other information, such as wireless connection strength or status, a malfunction within the smart necklace1000, a volume setting of the smart necklace1000, or the like.

FIG. 10Bis an enlarged view of the left portion1051of the smart necklace1000including indicators1035and one camera from the stereo camera pair1021according to an implementation of the present disclosure. As illustrated, the camera from the stereo camera pair1021is positioned distal to the band1081on the left portion1051. Towards the band1081from the camera from the stereo camera pair1021is the button1034. The indicator1035ais positioned towards the band1081from the button1034. The indicator1035bis positioned further the band1081from the indicator1035a, and the indicator1035cis positioned towards the band1081from the indicator1035b.

It may be desirable for the indicators1035to be adjacent. This allows the display of information to be central to a particular location, so that all display information can be gathered by looking at one location of the smart necklace1000. It may be desirable for the indicators1035to be not visible. They may become disturbing to others, such as during a movie at a movie theatre, at a nice restaurant, etc. To prevent this disturbance, the indicators1035may be positioned at a location where they are not visible to others. By grouping the indicators1035together, it may be easier to conceal the indicators1035.

FIG. 11is a view of a smart necklace with additional back-space for integrated electronics. The smart necklace1100corresponds to the smart necklace100, and has similar components as the smart necklace100, which are located within the left portion1151, the right portion1152, and the band1154. The components illustrated on the left portion1151may instead be positioned on the right portion1152, and the components illustrated on the right portion1152may instead be positioned on the left portion1151. Additional components that are not illustrated may be positioned on the smart necklace1100, and some of the components illustrated on the smart necklace1100may be removed.

The band1154may be enlarged as compared to other embodiments illustrated herein. The enlarged band1154may provide additional space for components to be positioned. The enlarged band1154may also provide additional comfort to the user. For some users, a skinny band may result in an uncomfortable amount of force being applied to a small area of the user's neck. By enlarging the band1154, more surface is provided such that the weight of the smart necklace1100is distributed across a larger portion of the user's neck.

The enlarged band1154also provides additional space to house components. This additional space can improve the weight distribution within the smart necklace1100. If most or all components were positioned on the right portion1152and the left portion1151and not the band1154, then most of the weight would be towards the front of the smart necklace1100. This may be uncomfortable for the users, and may result in too much swinging of the necklace. By distributing the weight between the front and the back, the smart necklace1100will be more comfortable to wear and will rest on the user's neck more easily.

As illustrated, the band1154includes a processor1111, a memory1112and a battery1141. Additional components may be positioned on the band as well. The components positioned on the band1154may be positioned such that the weight is evenly distributed about a center line that extends from the band1154through the middle of the left portion1151and the right portion1152.

The left portion1151may include a flattened surface that is distal to the band1154. The left portion1151includes stereo cameras121. Also included in the left portion1151is a left speaker1132b. The left portion1151also includes a Bluetooth interface1142A for facilitating wireless communications with any Bluetooth capable device, such as a cellphone, a laptop, a Bluetooth headset, etc. Also included in the left portion1151is an I/O port1143b. The I/O port1143bmay be any port capable of receiving and transmitting data, such as a USB port, a mini-USB port, an HDMI port, or the like.

The right portion1152includes a right speaker1123a. The right speaker1123amay communicate with the left speaker1123bsuch that the speakers can provide stereo audio. Also included is a camera1122. The camera1122may be a wide angle camera, such as one capable of capturing data across a 120 degree field of view.

The right portion1152(or in various embodiments, the left portion1151) may include a surface that is at least partially flattened. The surface may include buttons1134. Because the surface is flattened, user interaction with the buttons1134may be easier. In a preferred embodiment, the buttons1134may each correlate to a mode of operation of the smart necklace1100, such that when one of the buttons1134is pressed, the smart necklace1100operates in a particular mode. In various embodiments, a user may scan through the modes in other manners, such as repeated clicking of one of the buttons1134, repeatedly clicking one of the buttons1134to scroll in one direction and another of the buttons1134to scroll in the other, voice commands or the like.

Using the one-button-per-mode system provides simplified user input. In many situations, it may be preferred for the user to be able to switch modes without drawing attention to himself. By learning the location of the buttons1134and which mode of operation each button causes, the user can quietly and easily select a preferred operating mode without drawing attention of other people around him. Situations may also arise where it would be difficult for the smart necklace1100to understand the user's voice over the ambient noise. This one-button-per-mode system prevents this issue, as no speaking may be required of the user.

The smart necklace1100may operate in four modes: explorer mode, scan mode, find mode and capture. While in the explorer mode, the smart necklace1100provides data to the user associated with the surroundings of the user. In some embodiments, the smart necklace1100may describe data collected by the stereo cameras1121, the camera1122and/or any other sensor to the user. In some embodiments, the smart necklace1100may only described data that is collected while the user is moving (i.e., the field of view of the stereo cameras1121and/or the camera1122is changing). The data may only be certain data, such as hazards, whether a friend of the user is passing by, whether a user's favorite restaurant is detected, etc.

While in the scan mode, the smart necklace1100may describe everything that is in the field of view of the stereo cameras1121, the camera1122and/or any other sensor. For example, the smart necklace1100may describe everything in the field of view, such as by telling the user that object X is at your 10:00, object Y is at your 11:00, objects Z and W are at your 12:00, etc. The smart necklace1100may operate in the scan mode even if it is not in motion and/or being worn. For example, the user could place the smart necklace1100in a charging dock or in any other position in which the smart necklace1100could capture data with the stereo cameras1121and/or the camera1122. The smart necklace1100could then continue to describe information that is in the field of view of the stereo cameras1121and/or the camera1122.

While in the find mode, the smart necklace1100can navigate the user to a desired object, place, person, etc. The user can provide data about the desired object, place, person, etc., such as by speaking the name of the object, place, person, etc. The smart necklace1100can then determine the location of the object, place, person, etc. and provide navigation directions to the user.

The capture mode may allow the smart necklace1100to store its current position in the memory1112so that it can guide the user back to the same location at a later time. The capture mode may include2instructions—capture and return. Capture stores the position information (and possibly any obstacles that may arise during a return trip to the position) while return causes the smart necklace1100to provide navigation instructions to the user for a return to the position. In various embodiments, a single press of the capture button may indicate the capture instruction and a double click indicates the return instruction.

The right portion1152includes an I/O port1143a. The I/O port1143amay be any port capable of receiving and transmitting data, such as a USB port, a mini-USB port, an HDMI port, or the like. A strap1190may also be provided. In various embodiments, the strap1190may electrically connect the components positioned on the left portion1151to the components positioned on the right portion1152via the I/O ports1143. The electrical connection between the left portion1151and the right portion1152may increase robustness of the smart necklace1100. By allowing the components on the left portion1151to communicate with the components on the right portion1152via a shorter path (i.e., the strap1190), the components may become better synchronized. This may allow for faster data processing as well as more synchronized stereo output of the speakers1132and/or vibration units1133.

The strap1190may physically attach to the left portion1151and/or the right portion1152via a snap connector, a magnetic connection, a buckle or the like. This physical connection may increase stability of the smart necklace1100. The strap1190may act as a safety against the smart necklace1100falling off of the user and becoming damaged. The strap1190may also help to stabilize the smart necklace1100so that the smart necklace does not shift while the user is in motion.

In various embodiments, the strap1190may provide only a physical connection or only an electrical connection.

The smart necklace1100also includes an antenna1142. The antenna1142may be adapted to communicate wirelessly with another smart device1101, a cloud1105, a tablet or the like.

The smart device1101may include a processor1111A, a memory1112A and an antenna1142A. The antenna1142A may be adapted to communicate with the antenna1142of the smart necklace1100. The smart necklace1100may take advantage of the connection to the processor1111A and/or the memory1112A of the smart device1101. For example, the smart necklace1100may cause the processor1111A to perform some or all of the processing normally performed by the processor1111. Additionally, the smart necklace1100may use the memory1112A for storage instead of or in addition to the memory1112. In some embodiments, the smart necklace1100does not include the processor1111and/or the memory1112and relies solely on the remote device1101for processing and storage.

The cloud1105may include a processor1111B and a memory1112B. The antenna1142may be able to communicate with the cloud1105. The smart necklace1100may take advantage of the connection to the processor1111B and/or the memory1112B of the cloud1105. For example, the smart necklace1100may cause the processor1111B to perform some or all of the processing normally performed by the processor1111. Additionally, the smart necklace1100may use the memory1112B for storage instead of or in addition to the memory1112. In some embodiments, the smart necklace1100does not include the processor1111and/or the memory1112and relies solely on the cloud1105for processing and storage.

FIG. 12illustrates an exemplary method1200for human interaction using the smart necklace100. The method1200is only an example of the detection and analysis or processing of the first detected data and/or the second detected data. The blocks of method1200may be performed simultaneously or in various combinations of orders. In blocks1202and1204, the first data and the second data are detected using the camera122(including the stereo cameras121and/or the omnidirectional camera122) and/or additional sensor125. The degree to which data can be collected regarding the surrounding environment of the user and matters therein may depend on what or which camera122and/or additional sensor125are available and the processing limitations of the processor111and/or the external device and/or cloud. As such, the method1200may be adjusted accordingly in real time by monitoring such limitations.

The processor111may work in concert with the cameras122(including the stereo cameras121and/or the omnidirectional camera122) and/or additional sensor125for improving collection of the first detected data and/or the second detected data. The processor111may also consider whether the user or a person is requesting or attempting to convey information. For example, if a user is making a facial expression without speech to communicate with the smart necklace100, the processor111can direct the speaker132to pose follow-up questions or inquiries in order to supplement or clarify the detected data. For example, the method1200may direct an output speech to be generated, thereby asking the user to clarify the facial expression. The user may then respond in a voice command clarifying the conveyed information. In other embodiments, this facial expression recognition setup may be performed by a person other than the user.

In block1206, the method1200detects dynamic objects or beings. In one embodiment, the method1200may detect movement, changes in a scene or other dynamic regions as observed by cameras in order to focus the cameras122and/or additional sensor125on the detected dynamic regions. The processor111classifies the detected dynamic region as described below.

For example, detection of a person, living being, and/or a dynamic object may be performed by looking for changes in data detected by the cameras122and/or additional sensor125. Changes in data received from the cameras122and/or additional sensor125may be identified by first estimating the motion of the smart necklace100using the GPS124, the IMU123or techniques such as visual odometry which allow estimation of the motion of a camera by tracking corner or blob features between two camera frames. As such, the smart necklace100may identify motion in the surrounding environment of the user which does not correspond to the motion of the smart necklace100.

Upon identifying the changing parts of the scene within the first detected data and the second detected data, the smart necklace100seeks to recognize the changing elements, using techniques such as “eigenfaces” and “skeletal recognition” to recognize persons and faces. Additionally, standard techniques like Support Vector Machines, Deformable Parts Model and dynamic programming can be used to learn different models for various object/person classes. The types of features that can be used in these recognition tasks can be any combination of features like SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), Gist modeling, Sobel, Fast, and other features and/or techniques that enable the method1200to recognize a person, object, living being, or place/environment within a proximity of the user.

Thus, by detecting the dynamic regions, a new person entering the environment within the proximity of the user may be detected in block1212and a dynamic object or being can be detected in block1206and classified by the processor111accordingly. Simultaneously or thereafter, the smart necklace100extracts the remaining static regions in block1216. In one embodiment, in block1216, additional second detected data are collected from static objects in block1218and from an environment or place within a proximity of the user in block1220.

The microphone131may communicate with a speech recognition module to detect speech, conversation or interaction as shown in block1208. The smart necklace100may further extract conversation elements containing useful data in block1210. In block1222, the processor111matches extracted conversation or speech elements from block1210to the detected person, object, living being, place/environment, or combinations thereof.

In block1228, the method1200looks up prior relevant information or learned data based on context and based on the matched conversation events from block1222regarding the person, object, living being, place/environment, or combinations thereof. In block1226, the processor111stores relevant information for later use in the memory112based on prior relevant information. For example, if the processor111detects facial features of a person entering the environment and also detects that the new person is speaking, the speech elements can be matched with the new person. Speech data related to the person may be stored in the memory112for later use. The microphone131may include a 3-D microphone or a microphone array to localize the origin of the sound or voice. The smart necklace100can track and log data related to the person in order to supplement the first detected data. The method1200may actively and automatically output a second output data in block1250based on the matched conversation events to the corresponding person, place/environment, living beings, or combinations thereof of block1222and further based on the inferred context and interaction key points from block1224.

The processing of data (e.g., in blocks1206-1250) can be performed by continuously analyzing data gathered by the cameras122and/or additional sensor125in real time. The external device and/or cloud may be utilized due to restraints on the information storage capacity of the memory112, energy capacity challenges associated with processing using solely the processor111, and processing power limits of the processor111. However, in one embodiment, both on-board and off-board processing capabilities are utilized to prepare for events in which the on-board processing may be preferable (e.g., a poor connection in cloud communications) to ensure a minimal level of capability. For example, if the method1200is implemented in a neck worn device/platform that may not have sufficient capacity to perform the blocks described herein, the external device and/or cloud can provide assistance in sharing the load of the processing.

In block1230, the processor111may passively output, using the speaker132, a first output data upon an input/request received by the processor111or a predetermined or scheduled event stored in the memory112.

The processor111may further actively and automatically output, using the speaker132, a second output data based on the first detected data and/or the second detected data, the previously detected, processed, and stored first and/or second data, the pre-programmed algorithm stored in the memory112.

As discussed above, the interface array130communicates with the user or another person based on the detected data. The interface array130may communicate via the display135or a projection system in communication with the processor111. The display135or projection system may be positioned on a remote device, such as a cellular telephone wirelessly connected to the smart necklace100. The interface array may also communicate via the speaker132.

The output images/videos may be displayed using an LCD, an organic light emitting display, a plasma display, light-emitting diodes, or any other display mechanism for displaying the output images/videos.

FIG. 13Aillustrates an exemplary method for navigating a smart necklace100and/or providing helpful information to a user of the smart necklace100based on detected data.

Block1301refers to various methods of data collection using the smart necklace100.

In block1302, the smart necklace100may detect image data using the camera122. The image data may correspond to the surrounding environment, objects or living beings therein, the user, and/or other surrounding elements. For example, the image data may be associated with the shape of a room or objects within the room. As another example, the smart necklace100may detect image data including facial recognition data.

In block1303, an IMU123is coupled to the platform and configured to detect inertial measurement data corresponding to a positioning, velocity, or acceleration of the intelligent navigation device. A GPS unit (GPS)124is configured to detect location data corresponding to a location of the intelligent navigation device.

In block1304, speech data or audio data is detected using the microphone131. This information may be information associated with the user, with the environment, with other people, actions, events, and various other items regarding social interaction and the surrounding environment. For example, when in a particular room, the smart necklace100, via the interface array130, may receive information from the user or another person, such as the type of room (i.e., “this is the living room”). As another example, a user may want to name or add information regarding a particular person. In this instance, the smart necklace100, via the interface array130, may receive information from the user such as to name the person (i.e., “he is Johnny”). Alternatively, the processor111may actively infer this information by parsing a conversation with the other person, without a user input that is directed to the smart necklace100.

Referring to another example, the user may provide input to the smart necklace100that the user is performing a particular action, such as going to lunch. As another example, the user may provide input to the device that a hazard exists at the current position and a description of the hazard. Alternatively, the processor111may actively infer this information by parsing conversations, analyzing detected image data, etc. as discussed above with respect toFIG. 5and as discussed further below.

The data may further include a request. This request may be, for example, a request to identify a person, identify a room, identify an object, identify any other place, navigate to a certain location such as an address or a particular room in a building, to remind the user of his current action, what color an object is, if an outfit matches, where another person is pointing or looking, etc. The output of block1314(determining a desirable event or action) or of block1316(determining a destination) can be based on the requested information. Although speech data is discussed with respect to block1304, the information can be gathered using any combination of components of the interface array130and/or the sensor array120.

In block1305, the processor111may establish communication with a person other than the user via the interface array130and/or via a connection with a remote device. The remote connection may be established via a wireless communication antenna142, as discussed further below.

The processor111may determine whether communication with a person other than the user is desirable or requested. For example, when the detected data suggests that the user requires an opinion of another person, a communication channel may be established with a device of another person. For example, when the detected speech regarding an outfit of the user, facial recognition data regarding the user being indecisive or wondering about what to wear, and/or perceived action of a user in front of a mirror indicate that the user needs fashion advice from another person, a video teleconference between the user and a friend of the user may be established. From prior conversations/interactions, the processor111may have previously stored a user's friend's contact information. The processor111may categorize types of friends of the user and recognize that this communication needs to be with a friend that the user is comfortable with. The processor111may output data to the user letting the user know that a video conference or teleconference will be established with the friend. The smart necklace100may provide a video connection to a friend of the user or send a picture of the outfit to a friend of the user. In this example, the friend may provide a response as to whether or not the outfit matches. The friend may also assist the user in finding an alternate outfit that matches.

In block1306, data is received from the user, the environment, and/or other objects/beings via the interface array130. For example, data may be detected from a touch-screen display135, from a keyboard or buttons of an input device134, or other devices capable of interfacing with the user or another person to receive input data.

In block1307, data may be detected using other components of the sensor array120. For example, data may be detected from the other sensors125as discussed above. This information may be information associated with the user, with the environment, objects within the environment, and/or with other living beings/people.

In block1308, the smart necklace100may also receive data from another device using the antenna142or the I/O port, such as data regarding a map of a building, or any other data. Data may be shared among the smart necklace100, other devices of the user (for example, a portable electronic device of the user such as a smart phone or tablet), a remote server, or devices of others connected and with permission to access (for example, devices of the friends of the user).

In block1308, data is retrieved via the antenna142and/or I/O port143. This data may be information indicating to the smart necklace100that the user should be performing a particular set of actions. For example, the user may be in a hospital. The received information may be processed real time or stored for later use in the memory112. The information may relate to mealtimes of the user. The smart necklace100may then know that the user is to eat lunch at 12:00 pm in the cafeteria every day. As another example, the smart necklace100may access the user's cellular phone and download the user's schedule.

In block1311, the processor111matches collected data from block1301to relevant data stored in the memory. This includes object recognition. The processor111recognizes an object in the surrounding environment by analyzing the detected data based on the stored object data and at least one of the inertial measurement data or the location data. The object data stored in the memory112can be obtained from block1312.

In block1312, the memory112stores relevant data locally and/or remotely. For example, locally stored data may be data stored in a memory coupled to the smart necklace100. Remotely stored data may include data accessed from a remote server or another device via the antenna142and/or I/O port143. A schedule of the user may be periodically transmitted to the smart necklace100via the antenna142.

In block1313, the processor111matches data collected in block1301to relevant data stored in the memory. This includes object recognition as discussed above. The processor111recognizes an object in the surrounding environment by analyzing the detected data based on the stored object data and at least one of the inertial measurement data or the location data. The retrieved data can include data stored in the cloud or the internet. The processor111determines what information is desirable to process the request. For example, if the user requested to be navigated to the living room, the smart necklace100may need to know where the living room is, a layout of an entire route from the user to the living room and any hazards that may be present. As another example, if the user asked if his clothes match, then the smart necklace100may need to know what type of clothes match, what colors match and what the user is wearing.

The processor111accesses the memory112to retrieve the information desired to process the request. For example, if the user requested to be navigated to the living room, the smart necklace100may retrieve the location of the living room, a layout of the route from the user to the living room and any known hazards.

The processor111may determine whether or not the memory112has sufficient helpful information regarding the detected data. For example, when the user requests walking directions between two points, and a layout of a route is not available in the memory112, the smart necklace100may access the internet or the cloud via the antenna142and/or the I/O port143to retrieve this missing information.

In block1314, the processor111determines a desirable event or action. The processor111may determine a desirable event or action based on the recognized object, the previously determined user data and a current time or day. Current day or time is relevant for determining the current desirable event, action, destination, speech, etc. as discussed below.

The smart necklace100may determine whether or not the user should be at a particular location and/or performing a particular action at any given time. For example, the processor111may match a previously stored lunch event to a current date/time (i.e., noon). The processor111may also match the previously stored lunch event to a time before the event. For example, if the user is to have lunch at noon somewhere that is 30 minutes away, the processor may determine a match at 11:30 am. As another example, a desirable action may be to wear certain clothes with other matching items. A desirable event may be to go to lunch if the current day/time indicates that the user in a nursing home should attend a lunch gathering event.

In block1315, the smart necklace100, via the interface array130, may output data based on the inferred current desirable event, action/destination, etc. For example, if the inferred action is to find matching clothes, the processor may determine whether or not the outfit matches. As another example, if a destination is inferred, the processor may determine a viable navigation route for the user. The output may be, for example, a series of verbal phrases (i.e., step-by-step walking directions) via the speaker132. The output may also be, for example, vibrations informing the user of the data. For example, a vibration in the left side of the smart necklace100may signify to turn left, a vibration in the right side of the smart necklace100may signify to turn right, a vibration in both sides may signify to stop, a continuous vibration in both the right and the left sides of the smart necklace100may signify to slow down, or any other combination of vibrations may indicate any of these or any other command.

Discussion now turns to navigation features of the smart necklace100. In order to provide navigation information to the user, the processor111at least determines two sets of data: (I) data regarding positioning and/or location of the smart necklace100and/or the user and (II) data regarding the surrounding environment, persons, objects, living beings, etc.

Referring back to block1302, data regarding the surrounding terrain of the smart necklace100is detected using the camera122. As discussed above, the blocks inFIG. 13Aare not necessarily performed in the order shown. The processor111may determine, for example, that further image data is required to learn about the terrain after a destination is determined. When navigating indoors, the standalone GPS units may not provide enough information to a blind user to navigate around obstacles and reach desired locations or features. The smart necklace100may recognize, for instance, stairs, exits, and restrooms and appropriately store them in the memory112.

For example, the stereo cameras121may provide depth information of the surrounding environment and obstacles. Alternatively or in addition, one or more other cameras122may be utilized to provide information regarding the surrounding environment.

Referring back to block1303, data using the GPS124and/or the IMU123is detected. This data can be used along with data obtained from the camera122to gain an understanding of the terrain.

In blocks1317,1318a, and1318b, the processor111analyzes data obtained using the camera122based on the data obtained from the GPS124and/or the IMU123, and vice versa. In block1317, information set (II) can be used to gain a better/more accurate understanding of the information set (I) and vice versa.

In block1317, the processor determines data regarding the location or positioning of the smart necklace100using at least one of image data, inertial measurement data obtained using the IMU123, location data obtained using the GPS124, and relevant stored data (for example, map data stored in the memory112).

In block1318a, the processor111may analyze features of images collected using the camera122and recognize the environment objects using object recognition. For example, data collected by the IMU123can be used to determine the amount and speed of movement to improve accuracy of detection using data collected by the camera122. In addition, the IMU123may indicate a direction in which the collected information is located. For example, if the IMU123indicates that the information is regarding objects from a direction above the smart necklace100, the processor111can determine that the surface is more likely to be ceiling than ground.

In addition, data collected using the GPS124can enhance identification of data collected by the camera122. For example, if the camera122provides an image of the building, the processor111can determine if the building is detected correctly by utilizing data regarding the location of the user in the world, because building types differ in different parts of the world.

The GPS information may be inadequate because it may not provide sufficiently detailed information about the surrounding environment. However, the GPS information can be utilized along with visual data from the camera122to draw inferences that are helpful to the user. For example, if the GPS information indicates that the smart necklace100is currently inside a building, and the camera122provides information regarding an object, the processor111can limit its search to objects that would rationally be inside the building. For example, if an image provided by the camera122appears like a truck, the processor111can rule out the possibility that the object is a truck based on the GPS information. In other words, it is more likely to be an image of a poster of a truck, because the poster can rationally be within a building and a truck cannot. The GPS124provides location information, which along with the inertial guidance information, including velocity and orientation information provided by the IMU123, allows the processor111to help direct the user.

The memory112may store, for example, map information or data to help locate and provide navigation commands to the user. The map data may be preloaded, downloaded wirelessly through the antenna142, or may be visually determined, such as by capturing a building map posted near a building's entrance, or built from previous encounters and recordings. The map data may be abstract, such as a network diagram with edges, or a series of coordinates with features. The map data may contain points of interest to the user, and as the user walks, the camera122may passively recognize additional points of interest and update the map data.

In block1316, the processor111determines a desired destination based on the determined desirable action or event.

For example, the smart necklace100may direct the user to an empty seat, or may remember the user's specific seat in order to navigate the user away and subsequently return to the same seat. Other points of interest may be potential hazards, descriptions of surrounding structures, alternate routes, and other locations. Additional data and points of interest can be downloaded and/or uploaded to mobile devices and other devices, social networks, or the cloud, through Bluetooth or other wireless networks.

In block1318b, based on the analyzed data, a maneuverability condition/non-traversable region is detected. For example, a non-traversable region may be a region where the user cannot safely travel, such as a tar pit.

In block1319, the processor111determines a path over which the user can travel. The path excludes the detected non-traversable regions in block1318b. The smart necklace100may determine paths for navigation, which may be further modified for the user's needs. For example, a blind person may prefer routes that follow walls. Using the IMU123and/or the GPS124and other sensors, the smart necklace100can determine the user's location and orientation to guide them along the path, avoiding obstacles.

For example, the path may be towards a desired object (empty chair) as discussed above. The smart necklace100may identify obstacles or paths for the user, and based on either the speed of the traveler or the intended direction of the traveler, be able to filter down what the significant obstacles or potential paths are. The smart necklace100may then guide the user based on those significant obstacles or paths. Guidance may be, for example, auditory feedback or vibratory feedback, for either the path or objects to avoid.

In block1322, the output data from block1315may be conveyed to the user using various outputs of the interface array130. Multimode feedback is provided to the user to guide the user on the path. This feedback is also provided to guide the user towards the desired destination/object and is presented via a combination of speech, vibration, mechanical feedback, electrical stimulation, display, etc. With blind users, the processor111may keep the range of vision in mind when outputting information. A blind or partially blind person can identify most of the things that are less than three feet away using a cane. Objects and other items of interest more than 30 feet away may not be of utmost importance because of the distance.

While travelling along the path, the smart necklace100may inform the user about signs or hazards along the path. The vibration unit133and/or the speaker132provide audio and haptic cues to help guide the user along the path. For example, the speaker132may play a command to move forward a specified distance. Then, special audio tones or audio patterns can play when the user is at a waypoint, and guide the user to make a turn by providing additional tones or audio patterns. A first tone, audio pattern or vibration can alert the user to the start of a turn. For example, a single tone or a vibration from a left smart necklace may indicate a left turn. A second tone, audio pattern or vibration can alert the user that the turn is complete. For example, two tones may be provided, or the vibration may stop so that a left device ceases to vibrate, when the turn is complete.

Different tones or patterns may also signify different degrees of turns, such as a specific tone for a 45 degree turn and a specific tone for a 90 degree turn. Alternatively or in addition to tones and vibrations, the smart necklace100may provide verbal cues, similar to a car GPS navigation command. High level alerts may also be provided through audio feedback. For example, as the smart necklace100reaches a predetermined distance—such as a foot or other value which may be stored in the memory112and may be adjusted—from an obstacle or hazard, the speaker132and/or the vibration unit133may provide audible alerts. As the smart necklace100gets closer to the obstacle, the audible alerts and/or vibrations may increase in intensity or frequency.

As an example of the method illustrated inFIG. 6A, the user may give a voice command, “Take me to building X in Y campus.” The smart necklace100may then download or retrieve from memory a relevant map, or may navigate based on perceived images from the camera122. As the user follows the navigation commands from the smart necklace100, the user may walk by a coffee shop in the morning, and the smart necklace100would recognize the coffee shop. The smart necklace100may use this recognition and the time of day, along with the user's habits, and appropriately alert the user that the coffee shop is nearby. The smart necklace100may verbally alert the user through the speaker132. The user may use the input device134to adjust settings, which for example may control the types of alerts, what details to announce, and other parameters which may relate to object recognition or alert settings. The user may turn on or off certain features as needed.

FIG. 6Billustrates an exemplary method for providing assistance to a user of the smart necklace100based on an inferred current desirable event, action, destination, etc. In block1350, data is retrieved from the memory112and/or via the antenna142and/or the I/O port143. This data may be information indicating to the smart necklace100that the user should be performing a particular set of actions. For example, the user may be in a hospital. In this example, let's assume that the memory112includes information related to mealtimes of the user. The device may then know that the user is to eat lunch at 12:00 pm in the cafeteria every day. As another example, the user may inform the smart necklace100of his/her schedule. For example, the user may inform the device that he/she has a meeting at 5:00 PM this Friday at 600 Anton Blvd.

In block1352, the smart necklace100may determine whether or not the user should be at a particular location and/or performing a particular action at any given time. If not, the process may return to block1350. If so, the process may proceed to block1354.

In block1354, data associated with the current actions of the user is detected by the sensor array120. For example, the GPS124and/or the IMU123may sense that the smart necklace100is traveling towards the cafeteria or towards 600 Anton Blvd. As another example, the microphone131may detect data indicating that the user is busy talking to another person.

In block1356, it is determined whether the current actions of the user match the particular set of actions from block1350. For example, if the user is not moving towards the cafeteria and he is supposed be at lunch in 5 minutes, the current actions do not match the particular actions. If the current actions do match the particular actions from block1350, then the process returns to block1354to ensure that the user continues to perform the particular actions.

In block1358, the user is informed of the particular actions via the interface array130. For example, if the device is travelling away from the cafeteria, the smart necklace100may provide data to the user that he should be going to lunch now. If the user does not begin the particular set of actions retrieved in block1350, then the smart necklace100may again notify the user after a predetermined amount of time. However, the smart necklace100may have learned when the user does not want to be interrupted. For example, the user may not want to be interrupted during a conversation. If the microphone131detects that the user is having a conversation, the smart necklace100may wait until after the conversation to inform the user.

The smart necklace100may determine, via the interface array130, whether or not the user needs more information. The smart necklace100may have previously learned preferences of the user, such as if he is wobbling back and forth, he requires directions. The smart necklace100may also request navigational information. For example, the user may request directions to the cafeteria. The user can provide data to the smart necklace100via the interface array130that he does or does not need more information. If the user does not require additional information, the process proceeds to block1364.

In block1364, the processor111determines whether or not the particular set of actions from block1350is complete. If the particular actions are complete, then the process ends. If the particular actions are not complete, then the process returns to block1354.

An example of the process inFIG. 13Bwill now be provided. Assume that the user is supposed to be in the cafeteria for lunch. The smart necklace100may determine that the user is supposed to be in the cafeteria and that the user is not moving towards the cafeteria. The smart necklace100may then, via the interface array130, provide feedback to the user, such as audio feedback that says, “time to go to lunch.” The user may then return feedback to the smart necklace100, such as “I'm not ready yet” or “let's go.” If the user is not yet ready, the smart necklace100may remind the user again that it is lunch time after a predetermined amount of time. When he is ready, the user may provide data to the smart necklace100that he is ready to go.

The smart necklace100may then request feedback from the user to determine whether he needs directions or not. If the user responds no, then the smart necklace100may not provide any current information. However, if the user responds yes, then the smart necklace100may navigate the user to the cafeteria. The smart necklace100may also be configured so that it communicates with another device. For example, the smart necklace100may provide data to a terminal in the cafeteria that the user is on his way for lunch, so that his food can be prepared and a seat ready for him when he arrives. As the user arrives, the smart necklace100may provide additional data to the user such as where his table is and who he is sitting with.

In some embodiments, the smart necklace100may learn preferences of the user. For example, it may learn what chair the user normally sits at for lunch. In this example, the smart necklace100may determine where the user's normal lunchtime chair is. To do so, it may use sensed visual data from the camera122, position data from the GPS124, the IMU123and/or any other detected data, as well as shape analysis as described above in regards toFIG. 2. Once the smart necklace100determines where the chair is, it may guide the user to the chair so that the user can safely get to the chair and sit down.

FIG. 14illustrates an exemplary method of danger assistance by the smart necklace100. In block1400, data is detected by the sensor array120. This data may be visual data, position data, or any other data that the sensor array can sense. For example, the IMU123may detect data indicating that the user has fallen down or that the user is in an upside down position. The camera122may detect visual data such as a large object extremely close to the lens. The sensor array120may detect any other information such as data indicating a fire or a flood.

In block1402, the detected data is compared to data indicative of danger stored in the memory112. This data in the memory112may be, for example, data associated with a falling motion of a user. The data may also be, for example, data associated with an object falling on the user. The processor111compares the data indicative of danger from the memory112to the detected data to determine if there is a match. For example, if detected data matches data indicating that a large object has fallen on the user, then a match exists.

In block1404, if no match exists between the data indicative of danger and the detected data, then the process returns to block1400where data continues to be detected. If, however, a match does exist, then the process proceeds to block1406. In block1406, the smart necklace100requests that the user provide information such as whether the user is ok. The user may provide this information via the interface array130. This information may be, for example, a spoken “yes, I am ok,” or “no, I am not ok.”

In block1408, it is determined whether the user responded that he needs assistance, responded that he does not need assistance, or did not respond at all. If the user responded that he does not need assistance, then the process returns to block1400. If the user responded that he does need assistance or if the user did not respond to the inquiry, then the process proceeds to block1410.

In block1410, the smart necklace100may, via the interface array130, request that the user provide information about whether the user wants to communicate with a person or to have a remote device alerted.

In block1412, it is determined whether or not the user selected to communicate with a person. If the user selected to communicate with a person, then in block1414, a communication channel may be established between the user and the desired people/person. Additionally, the user may select whom he wants to speak with. For example, he may wish to contact his personal physician, the police, a friend, or any other person or service. The smart necklace100may also have learned with whom to open a communication channel. For example, if fire data is detected, the smart necklace100may open a communication with a fire department or “911” call center.

The communication may be established, for example, by connecting the smart necklace to a cellular device via the antenna142and/or the I/O port143. After the connection is established, the smart necklace100may cause the cellular device to place a video call or a voice call to the requested person or institution. The microphone131of the smart necklace100may act as the microphone for the cellular device and the speaker132of the smart necklace100may act as the speaker of the cellular device. Once the communication is established, the user may communicate with the requested person and provide information. The smart necklace100may also provide information to a device on the other end of the communication, such as any data associated with the danger, any location data, etc. Any information may also be communicated via a Wi-Fi, Bluetooth, etc. element of the smart necklace100. For example, the smart necklace100may establish a VoIP connection via Wi-Fi.

If, in block1412, the user did not select to communicate with a person, or the user did not respond, the process may proceed to block1416. In block1416, a remote device is alerted of the danger via the antenna and/or the I/O port. This alert may consist of any data captured around the time of the incident, any location data, etc. The alert may be communicated by a connection to a cellular device via the antenna142and/or the I/O port143, Wi-Fi, Bluetooth, etc.

FIG. 15Aillustrates an exemplary method for safety monitoring and alerting. The process begins in block1500when movement is detected by the smart necklace100. This movement may be detected, for example, by the IMU123or the GPS124. After the process begins, in block1502, new data is detected by the smart necklace using the sensor array120. This data may include any data detected by the sensor array120, such as visual data (streets, sidewalks, people), position data (location of the user, direction of travel), audio data (such as a moving car sound, a siren, an alarm), or any other data. In block1506, this new data is stored in the memory112.

In block1506, this new data is compared to data recorded in the memory112. The data recorded in the memory112may include data captured at the same location at a previous time. The data recorded in the memory112may also include data captured at the same location or nearby at a time prior to the current time, such as milliseconds, seconds, or minutes prior to the current time. The processor may make this comparison using the object recognition method ofFIG. 2.

In block1508, it is determined whether or not a divergence is detected between the new data and the data recorded in the memory112. This divergence may include, for example, data indicating that a new object is in the field of view of the smart necklace100that was not previously detected. A divergence may also include, for example, that a previously-present object is no longer in the field of view. The divergence may also include, for example, a new sound, such as a police siren.

In block1510, if a divergence is detected in the new data, a second data may be output to the user via the interface array130based on this divergence. As an example, let's assume that a divergence includes a boulder in the middle of the sidewalk. In this example, the smart necklace100may provide data to the user indicating the divergence. For example, the smart necklace100may, using the speaker132, inform the user that an object matching the shape of a boulder is 10 feet directly ahead of the user. Alternately or additionally, the smart necklace100may provide haptic feedback to the user based on the divergence.

In block1512, it is determined whether or not the divergence includes text. For example, the divergence may be a sign or police tape including text. The processor111may make this determination by comparing the new detected data to data indicating the shape of text characters to determine if any matches exist.

In block1514, it is determined that the divergence includes text, then the smart necklace100may output data to the user via the interface array130based on the text of the divergence. For example, the data may include audio data indicating the content of the text of the divergence.

In block1516, it is determined whether or not the divergence presents a hazard. The memory112may have stored data which can be compared to detected data in order to determine if a divergence is hazardous. For example, the memory112may have stored therein visual data associated with a bobcat and an indicator that a bobcat is hazardous. As another example, the memory112may have stored therein visual data associated with the shape of a caution sign and the word caution, and that this data is an indicator of a hazard.

As another example, instead of having data representing a dangerous object, the memory112may store situational data. An example of situational data is that the smart necklace100may recognize that if a large object is in the middle of a sidewalk that the user is walking along, the object may present a hazard. Another example of situational data is that the smart necklace may recognize that if visual data of an area had been previously sensed and stored, and the visual data of the area is significantly different in the present sensing of the area, then a danger may exist.

In block1518, if it is determined that the divergence does present a hazard, the smart necklace100may warn the user via the interface array130. The warning may include, for example, an output via the speaker132informing the user that a hazard exists and the location of the hazard. The smart necklace100may also output, for example, the type of hazard. If the smart necklace100detected a caution sign that read “potholes in the sidewalk,” then the smart necklace100may output data to the user informing the user that potholes exist in the sidewalk. The smart necklace100may also inform the user of the locations of the potholes as the smart necklace100detects them. In some embodiments, the smart necklace100may provide vibrational data to the user via the vibration unit133. For example, as the user approaches a hazard, the vibration unit133may produce increasingly frequent vibrations.

Next, it is determined whether or not more movement of the smart necklace100is detected. If no more movement is detected, then the user is standing still and the process may end. If movement is detected, then the process may return to block1502. If the process ends because of a lack of movement, it can restart at any time by detection of movement of the smart necklace100.

FIG. 15Billustrates an example of the method ofFIG. 15A. In frame1550ofFIG. 15B, a user1556is walking along a sidewalk. Let's assume that the user has previously walked along the sidewalk and the memory112has stored therein data detected by the sensor array120during the previous trips along the sidewalk.

In frame1552, a hazard1558is present on the sidewalk in the direction of travel of the user. The hazard includes a caution sign1560as well as two caution cones1562. When the hazard is in the field of view of the smart necklace100, the smart necklace100may compare the detected data to stored data. The processor111may then determine that the hazard is a divergence and it may provide data to the user1556based on the divergence. The data may include, for example, a description of the two cones1562, the fact that a sign is present and any other large diversions. The smart necklace100may also detect that the diversion includes text. The smart necklace100may provide the user1556data based on the text, such as reading the text to the user.

The smart necklace100may have learned that a caution sign1560or a caution cone1562presents a hazard, and determine that the caution sign1560and/or the caution cone1562present a hazard. The smart necklace100may also determine this by identifying that the divergence is significant. The smart necklace100may use the fact that the caution sign1560and/or cones1562are positioned in a direction of travel of the user1556to determine that they present a hazard. The smart necklace100may then provide data to the user1556indicating that the hazard1558is present. The smart necklace100may, for example, output the type and/or location of the hazard1558to the user1556using the speaker132. The smart necklace100may also, for example, vibrate with increasing frequency as the user approaches the hazard1558.

For example, in frame1554, the user may utilize the output from the smart necklace100to navigate around the hazard1558. If the smart necklace100includes a vibration unit133, then the smart necklace100may vibrate more on one side than another to indicate the location of the hazard1558. In frame1554, the hazard is more to the right side of the user1556than the left side. So, the smart necklace100may vibrate on the right side and not on the left, indicating that the hazard is on the right. Alternately, the smart necklace100may vibrate on the left side and not the right, indicating that it is safer on the left side.

FIG. 16Aillustrates an exemplary method for providing navigation assistance to the user. This method may be used, for example, to inform a user when he should move forward in line. It may also be used, for example, to inform a user of when he should slow down or speed up based on a another person. This method may also be used, for example, to inform a user of when he should slow down or speed up so that he can walk at the same pace as his friends.

In block1600, the smart necklace100, using the sensor array120, detects a distance from the smart necklace to a moving object, such as a person. The object can be in any relative position to the user so long as it is in the field of view of the camera122. The smart necklace may also, in some embodiments, determine a speed at which the object is moving.

In block1602, the smart necklace100may determine a speed at which the user is traveling, using the sensor array120. This speed may include a walking or running speed, or it may be a zero speed. Using the speed data, as well as other data, the processor111may determine a set of actions that the user is performing. For example, the user could be walking with friends. He also could be waiting in a line, or walking down the street in a hurry to get to a meeting. The processor may utilize the detected data to determine the action of the user.

In block1604, the processor111compares the speed and other data to data in the memory112. The data in the memory112may, for example, associate speed and other data to a preferred distance of the user to an object. For example, if the user is walking with friends, the preferred distance may be relatively short. As another example, if the user is rushing to a meeting, the preferred distance may be relative long so that the user does not accidentally run into the object.

In block1606, it is determined whether or not the detected distance is less than the preferred distance. This may be useful, for example, when the user is walking down the street faster than another person, or has prematurely started walking forward when waiting in a line.

If the detected distance is less than the preferred distance, then the method proceeds to block1608. In block1608, the smart necklace100outputs data to the user, via the interface array130, indicating that the detected distance is less than the preferred distance. This data may include, for example, audio data or haptic data. For example, the smart necklace100may vibrate with increasing frequency as the user approaches the object.

In block1610, it is determined whether the detected distance is greater than the preferred distance. This may be useful, for example, if the user is walking with friends and is moving at a faster pace.

If the detected distance is greater than the preferred distance, then the process proceeds to block1612. In block1612, the smart necklace100outputs data to the user, via the interface array130, indicating that the detected distance is greater than the preferred distance. This data may include, for example, audio data or haptic data. For example, the smart necklace100may vibrate with increasing intensity as the distance between the user and the object grows. In some embodiments, there are only certain situations in which the smart necklace100will alert the user that the distance is greater than the preferred distance. For example, if the user is walking alone, he will not care if he is far away from other persons and the smart necklace may not provide any data.

FIG. 16Billustrates an exemplary use of the method ofFIG. 16A. In frame1650, the user1656is walking down a sidewalk. The user is behind a person1658. At first, the user1656and the person1658may be traveling at the same speed. Initially, because of the same speeds, the distance from the user1656to the person958may be the preferred distance.

As illustrated in frame952, eventually the person958may slow down. In this situation, the detected distance from the user956to the person958may become less than the preferred distance. The smart necklace100may provide data to the user956including information that the detected distance is shorter than the preferred distance. The user956may then slow down based on the data from the smart necklace100. By receiving the data from the smart necklace100, the user956may slow until he is at the preferred distance from the person958, as illustrated in frame954.

FIG. 17illustrates an exemplary method for handling an obstruction of a camera122(including stereo cameras121). In reference toFIG. 11, an obstruction can be any occurrence that obstructs the view of either of the stereo cameras1121or the camera1122. For example, a lock of hair or an article of clothing can block the lens of any camera. The smart necklace1100can respond to the obstructed camera using the following method.

In block1700, it is determined whether or not an obstruction is detected. Assuming that the camera1122is obstructed, the obstruction may be detected in multiple fashions. Image data from the stereo cameras1121may be checked against data from the camera1122. If the data does not correlate, then the smart necklace1100may determine whether the data from the camera1122or the stereo cameras1121are more likely data that represents the actual information surrounding the user. The camera1122may also be adapted to determine the distance to an object. If an object is within a certain range, such as 3 inches, of the camera1122, then the smart necklace1100may determine that an obstruction is occurring. The stereo cameras1121may also be able to capture data near the camera1122and determine that an object is obstructing the view of the camera1122.

If an obstruction is not detected, then the method may be placed on hold until an obstruction is detected. If an obstruction is detected, then the smart necklace1100may determine, in block1702, which camera is obstructed. In this example, camera1122is obstructed. This determination may be made based on the result of the selected method for determining whether an obstruction has occurred in block1700.

In block1704, the smart necklace1100ignores data from the obstructed camera (the camera1122). If the smart necklace1100is using data collected from the obstructed camera1122as input to functions, then the functions may provide bad output. Because the stereo cameras1121are present on the left portion1151, the smart necklace1100can still provide data to the user based on image data captured by the stereo cameras1121.

In block1706, the smart necklace1100may alert the user of the obstruction. For example, the smart necklace1100may provide a spoken alert to the user that the camera1122is obstructed. In various embodiments, the smart necklace1100may play a specialized tone or vibration instructing the user of the obstruction and/or the location of the obstruction (such as a vibration on the right portion1152indicating an obstructed camera on the right portion1152).

In block1708, it is determined whether or not the obstruction has been removed. This determination can be made using the same techniques discussed above in reference to block1700. If the obstruction is not removed, then the method may remain in this state until the obstruction is removed.

If the obstruction is removed, then the smart necklace1100may stop ignoring data from the camera1122in block1710.

FIG. 18Aillustrates an exemplary method for determining the location of a desired object. In block1800, the smart necklace100may determine that a particular object or objects are desired. This may be determined with or without explicit instructions from the user.

For example, the user may be able to click a button while directing the camera122at an object to instruct the smart necklace100to find the object. The user may also be able to verbally instruct the smart necklace100to find the object. For example, the user may direct the camera122or the stereo cameras121towards an object and say “I want to find this.” The camera122may then capture image data of the desired object. The smart necklace100may detect movement via the IMU123. A certain movement of the smart necklace100may indicate that the user desires to find a particular object. The user may also give a command that includes a description of the object. For example, the user could speak the words “find me a box of X cereal” into the microphone131. The smart necklace100may then be able to recall data from the memory112about the object or retrieve data via the antenna142or the I/O port143about the object. Data associated with the object may be stored in the memory112until the smart necklace100locates the object.

In block1802, the smart necklace100may scan its field of view1990for the object or objects using the camera122, the stereo cameras121or any sensor125. The smart necklace100may continuously scan the field of view1990for the object or objects or the smart necklace100may be told when to search for the object. For example, the user could make a grocery list at home and scan each of the objects that the user wants to retrieve from the grocery store. It would be a waste of processing power for the smart necklace100to scan the field of view1990within the user's residence for the object or objects. In this situation, the smart necklace100may determine when it is in the grocery store where it is reasonable to scan for the objects.

In block1804, it is determined whether the object or objects have been detected. The smart necklace100may compare the object data in the memory112to data sensed by any component in the sensor array120. If the stored data does not match the sensed data, then the process returns to block1800. If the object is detected, then the smart necklace100may indicate that the desired object is detected. This indication may be in the form of audio output via the speaker132or via the vibration unit133. The smart necklace100may, for example, vibrate once when it detects the object. The smart necklace100may also output a beep or an audible instruction that the object has been detected.

In block1808, the clip provides data indicating the exact location of the desired object. For example, the vibration unit133may provide vibration or an audio tone with increasing frequency as the user approaches the object. The speaker132may also provide precise directions to the user about the location of the object.

FIG. 18Billustrates an exemplary use of the method ofFIG. 18A. In frame1850, the user1856is scanning a box of cereal1858with the smart necklace100. The box of cereal1858may be empty, and thus the user1856wants to replace the box of cereal1858. The user1856may be using the stereo cameras121of the smart necklace100to detect data from the box of cereal, such as the name or a barcode of the box of cereal1858.

In frame1852, the user1856is walking down an aisle in a supermarket. The smart necklace100may be scanning the field of view1990of the camera122and the stereo cameras121for the box of cereal1858. As the smart necklace100detects the box of cereal1858, it indicates this to the user1856. For example, the smart necklace100may provide a beep or a vibration.

In frame1854, the smart necklace100is directing the user1856to the precise location of the box of cereal1858. The smart necklace100may provide vibrations or tones of increasing frequency as the user1856approaches the box of cereal1858. The smart necklace100can also provide any other type of output to the user1856that would direct the user1856to the precise location of the box of cereal1858.

The discussion now turns to advantageous features and process of the present invention for providing output data to a user based on identified characters or images of a document. The smart necklace can perform optical character or image recognition on documents that a blind or low vision person is interested in reading while seated at or standing by a table, desk, counter, or the like. The smart necklace allows the person to keep his/her hands free. The smart necklace allows the person to interact with documents in a very portable manner. The smart necklace provides a portable optical character or image recognition solution that can be worn comfortably while the person reads a document on a table in front of the person. The person does not have to use his/her hands to hold the document. The smart necklace includes a camera that can be automatically directed towards the document in front of the person, scan the document, and perform optical character or image recognition on the document.

FIG. 19is a view of a smart necklace100used for providing output data to a user1956based on identified characters or images of a document1992. The smart necklace100may have any of the features of the smart necklace100,500,600,700,800,900,1000, and/or1100discussed above with respect toFIGS. 1-18B. The user1956may press a button of the smart necklace100(e.g., button1034ofFIG. 10discussed above) to begin learning about a document1992and its contents. Alternatively, the user1956may utter an audio command to begin the optical character or image recognition process. The audio command can be detected using the microphone131.

The smart necklace100scans or images the document1992using image data provided by the stereo cameras121. “Document” as used herein refers to any book, page, magazine, picture, or the like having characters or images having information that the user1956may be interested in. A document may also be a page or an image displayed electronically. That is, images or characters displayed on a tablet computer may be recognized similar to recognizing images or characters of paper documents.

The smart necklace100communicates to the user1956the contents of the document1992. The speaker132can audibly read the contents of the document1992to the user1956. The smart necklace100can also identify where the document1992needs to be signed. This feature allows the user1956to keep both hands free. That is, one hand can be used to hold the document1992in place and the other hand can be used to sign the document1992.

FIG. 20is a method of operating the smart necklace100to provide output data to the user1956based on identified characters or images of a document1992. In block2002, the smart necklace100automatically detects a document of interest (e.g., document1992shown inFIG. 19). Detection of documents and the optical character or image recognition process may be automatically activated depending on data captured using the sensor array120. This includes image data captured about the surrounding environments and recognized objects therein. Scanning of the environment and the objects therein was discussed above with respect toFIGS. 3A-3C, 6D, and 12-16B. The processor111may automatically detect a document of interest, and determine that the user1956would like to read the document1992or have information about the type of the document1992. The following example illustrates an application of the foregoing method. If a user1956comes home, the processor111may recognize the home environment based on GPS location data and previously recorded images corresponding to the home environment. The processor111can detect that certain mail documents and envelopes are positioned on the table1994. The processor111would recognize that the user1956is sitting down or standing next to the table1994. As a result, the processor111determines that the user1956is interested in learning about the mail documents. The processor111performs optical character or image recognition on the mail documents, and provides helpful information about the mail documents to the user1956.

The initial static optical angle of the stereo cameras121may or may not be facing downwards initially. When scanning the environment without a focus on documents, the static optical angle of the stereo cameras121may be forward-facing. As a result, the processor111may not have sufficient image data to recognize the document1992and its characters/images. The present invention advantageously adjusts the field of view1990of the stereo cameras121to accommodate document reading.

In block2012, the processor111adjusts the field of view1990of the stereo camera121to be downward, thereby covering the document1992. One or more mechanical rotating devices1996may be provided to assist in adjusting the field of view1990of the stereo cameras121. As shown inFIG. 19, a pair of mechanical rotating devices1996may be connected to the pair of stereo cameras121, respectively. The processor111is configured to control the mechanical rotating device1996to rotate the stereo cameras121. Exemplary rotation arrows1991are shown to illustrate some of the possible directions that the stereo cameras121may be turned. As a result of the rotation, the adjusted field of view1990faces downward and covers the recognized document1992.FIG. 19shows the field of view1990after downward adjustment to focus on the document1992. Prior to the adjustment, the field of view1990may have been facing forward to scan the environment.

The mechanical rotating device1996may include motors. The processor111or a controller connected to the processor111can control the mechanical rotating device1996. The motors can rotate the stereo cameras121downward in response to signals received from the processor111or the controller. A static friction device may also be utilized to maintain positioning of the stereo cameras121after rotation. Alternatively or in addition, a mechanical release device may be utilized for allowing the stereo cameras121to switch between different positions for adjusting the field of view1990. In the mail-reading example discussed above, the processor111may rotate the stereo cameras121to detect the table1994in front of the user1956. The stereo cameras121may then be moved to focus on the mail documents as the user1956flips through the mail.

The dynamic field of view1990of the stereo cameras121advantageously allows identifying a document of interest and providing helpful information about the document to the user1956. As shown inFIG. 19, the document1992can be detected and helpful information can be provided while the user1956is standing in front of the table1994or standing at the table1994. This obviates the discomfort of holding the document1992in front of the stereo cameras121. This can be particularly advantageous in applications that are meant to assist users with impaired vision. The user1956would need both hands to sign the document1992. By obviating the need to hold the document1992, comfort of the user1956is enhanced in writing on or signing the document1992.

Alternatively, the document1992may be detected in response to an input received from the user1956. In block2004, the smart necklace100receives an input from the user1956, requesting information about surrounding documents. The processor111adjusts the field of view1990, as described above with respect to block2012.

Alternatively, as shown in block2006, the document1992may be detected in response to an input received from the user1956. The input may request that the environment be scanned or explored. A scan mode or explorer mode may be selected, as discussed above with respect toFIG. 11. The processor111scans the environment and locates the document1992on the table1994.

In block2010, a document of interest is detected within the environment. The document1992can be detected by recognizing the corners of the page and discriminating them from the background (e.g., the table1994). The document1992can be distinguished based on color contrast between the document1992and the surrounding objects/environment (e.g., the table1994). The processor111adjusts the field of view1990, as described above with respect to block2012.

There may be more than one document in the field of view1990of the stereo cameras121. The processor111may select between the documents. The processor may further interact with the user1956to select a document, as discussed in further details below with respect to block2020.

A document-reading mode may also be provided. Upon selection of the mode by the user1956, the processor111provides information to the user1956about documents within the field of view1990of the stereo cameras121. A document description may also be provided such that general descriptions of documents are provided. As the user1956enters a coffee shop, the output information may distinguish between types of documents, such as advertisement papers versus news magazines.

In block2014, the processor111detects image data within the adjusted field of view1990of the stereo camera121. The image data corresponds to the document1992and its characters and images.

In block2016, the processor111is configured to analyze the detected image data using optical character or image recognition software program stored in the memory112. Characters or images of the document1992can be recognized using the optical character or image recognition software program. The recognized image may include a logo or a picture on the document1992. The characters can be recognized and analyzed in conjunction with another. The processor111can infer information that the document1992conveys by analyzing the relationship between the recognized characters and/or images. The processor111can analyze sentences and paragraphs of the document1992. The processor111can analyze the picture using object recognition as discussed above with respect toFIGS. 3A-3C, 6D, 12, 15A and 15B. The processor111may also utilize a translation program to translate text in foreign language to the preferred language of the user1956.

In block2018, the processor111determines output data based on the analyzed image data. The output data can be based on the information inferred from the recognized characters or images. In block2020, data is outputted to the user1956using the interface array130. The output data can be provided via an audio output using the speaker132. For a user with visual impairment, the output information may indicate what documents are there on the table and how close they are if they aligned. The processor111may ask the user1956via the speaker132about type of information that the user1956is interested in knowing. The processor111may adjust the output information based on user feedback (received, for example, via the microphone131).

Vibration feedback can also be provided. As discussed above with respect toFIG. 1, the vibration unit133may include a left vibration motor in the left portion151of the smart necklace100, and a right vibration motor in the right portion of the smart necklace100. This advantageously allows various combinations of haptic feedback using a left-side vibration that may differ from a right-side vibration. The vibration output information can be based on the determine output data in block2018.

When multiple documents are available in the surrounding environment, the processor111may interact with the user1956and determine which document the user1956is interested in reading. The processor111may recognize, for example, the first word of a document or a portion of the document that is highlighted (such as a logo). The processor111may inform the user1956about the types of the documents based on the first word or the logo. If there is a picture on the document, the processor111may generate output information descriptive of the picture. This brief general description would allow the user1956to decide whether he/she is interested in reading the entire document, or learning more about the document. The user1956may provide feedback (e.g., audio feedback or via the buttons) as to which document, if any, the user is interested in reading.

Block2020may be interactive and based on feedback provided by the user1956. Output information would be also based on the user feedback (as shown in block2006). The processor111may ask the user1956whether he/she is interested in reading the entire document1992or interested in knowing general information about the document1992. The user1956may then provide feedback or via pushing a button of the smart necklace100or via the microphone131.

The processor111determines that one or more of the stereo cameras121is obstructed. The speaker132is configured to output audio information to the user1956, alerting him/her about the obstruction. This would encourage the user1956to remove the obstruction. Alternatively or in addition, the alert is provided to the user1956using the vibration unit133.

In block2024, the processor111may store the output data of block2018in the memory112. The processor111may also store the output data in cloud memory. The processor111may selectively store information recognized about the documents. The processor111may determine that certain information will be useful to the user1956in future based on user feedback or based on previously known information about the user1956. The user1956may provide an audio command to the processor111via the microphone131, requesting certain information about the document1992to be stored in the memory112.

Alternatively or in addition, the processor111may automatically determine, without a current input from the user1956, that certain information about the document1992will be helpful in future. The following example illustrates an application of the foregoing process. The processor111may know that the user1956is interested in purchasing a certain type of laptop computer based on prior speech recognition, or based on recognized environmental data about the user1956. If the document1992includes an advertisement for the particular type of laptop, the processor111may remind the user1956about his/her interest in the laptop. The processor111may further store the advertisement information in the memory112. The processor111reminds the user1956in future about the advertisement information when the processor111determines based on GPS location data that the user1956is in proximity of a store offering the particular laptop at the advertised price. This process can be incorporated with the process discussed above with respect toFIG. 12such that inferred context and interaction key points in block1224would be based on previously or currently learned information from detected documents.

In block2026, the processor111may share the output data or learned information about the document1992via the antenna142. The processor111may email the information to an email address, as requested by the user1956. The processor111may share the output data with another portable electronic device of the user1956.

In block2022, the processor111re-adjusts the field of view1990of the stereo camera121based on object recognition. When a portion of the document1992exits the field of view1990, the field of view1990can be adjusted to focus on the document1992. The processor111may detect that the table1994is rotated, and adjust the field of view1990of the stereo cameras121accordingly.

The processor111may further provide output information to the user1956when the document1992cannot be read. If the document1992is only partially in view, the processor111may output information indicating that the smart necklace100can only see part of the document1992. The processor111can ask the user1956to realign the document1992. It may also ask that the user1956reposition himself or herself because that would affect the field of view1990. The processor111may minimize burden on user1956when the field of view1990can be adjusted using mechanical rotation or by focusing on a section of the image data within the field of view1990, without physical re-positioning of the document1992or the user1956.

In the preferred embodiments discussed above with respect toFIGS. 19 and 20, the smart necklace100includes two pairs of stereo cameras121, which may be positioned on either side of the user's neck. Stereo cameras provide depth information in both indoor and outdoor environments. The stereo cameras121may face forward, in front of the user1956, to establish a field of view1990. Alternatively, the pair of stereo cameras121may be positioned on the left portion151or the right portion152of the smart necklace100, for example, as discussed above with respect toFIG. 8.

In the preferred embodiments discussed above with respect toFIGS. 19 and 20, a pair of stereo cameras121is utilized in part. They stereo cameras121advantageously provide depth information. In addition to or instead of the stereo cameras121, one or more cameras122may be utilized, as discussed above with respect toFIGS. 1-11. The one or more cameras122may include an omni-directional camera. The one or more cameras122may be wide-angle cameras having at least 120 degrees of field of view1990. The wide-angle cameras122may be utilized in addition to the stereo cameras121to increase the field of view1990of the smart necklace100. The cameras122may also be placed where needed, such as behind the user's neck to provide data for an area behind the user1956.

In the embodiments discussed above with respect toFIGS. 19 and 20, the field of view1990of the stereo cameras121were changed via mechanical rotation. Alternatively, a first camera may be provided that faces forward and a second camera may be provided that faces downwards for document-reading applications. In such a configuration, less adjustment is needed because one camera is always facing downwards. This depends on design concerns such as whether it is power efficient or physically possible to fit an additional camera just for the downward document-reading purpose.

Alternatively, a camera or a set of cameras may be provided with a field of view1990that covers forward area and downward area. In this embodiment, mechanical adjustment of the camera may not be needed. Rather, the digital focusing for optical character or image recognition processing can be adjusted.

More particularly, the camera or set of cameras122provide video image data covering a broad area. The camera or set of cameras may include an omni-directional camera or a wide-angle camera, as discussed above with respect toFIGS. 1, 5A-6C and 7A-11. Instead of or in addition to mechanical focusing and rotation discussed above, the processor111may change the focus of processing. When the document1992is underneath the cameras, the downward sub-section of the video image data is processed.

The cameras utilized for the document-reading application discussed inFIGS. 19 and 20preferably have a non-uniform shape. This allows the user1956to feel the stereo cameras121and recognize the direction that they are facing. The stereo cameras121may be in shape of a cube. The stereo cameras121may have raised indents or braille to show direction of the stereo cameras121to a blind user. The cameras may be physically adjustable or rotatable to allow the user1956. This allows the user1956to feel the direction of the stereo cameras121and manually change its field of view1990by his/her fingers.

FIG. 21is a method of operating the smart necklace100to provide output data to the user1956based on identified characters or images of a document. In block2102, the processor111detects using at least one camera, image data corresponding to a surrounding environment. The at least one camera may be a pair of stereo cameras121, for example, as shown inFIG. 19. In addition or alternatively, the at least one camera may include a wide-angle camera and/or an omni-directional camera.

In block2104, the processor111detects a document positioned on a flat surface in the surrounding environment. The document1992may be positioned on a table1994, or any other flat surfaces such as a desk, counter, etc.

In block2106, the processor111adjusts the field of view1990of the at least one camera such that the detected document1992is within the adjusted field of view1990, as discussed above with respect to block2012ofFIG. 20.

In block2108, the processor111recognizes, using the optical character or image recognition processing data, at least one of a character or an image of the document1992. The image may be a logo having information about general characteristics of the document1992. The image may be a picture shown on the document1992.

In block2110, the processor111determines output data based on the recognized at character or the image of the document1992. In block2112, the processor111outputs data to the user1956regarding the document1992using the interface array130, for example, as discussed above with respect to block2020ofFIG. 20. Information inferred from the combination of the recognized characters and/or images may be communicated to the user1956via the speaker132, the vibration unit132, and/or the display135.

The optical character or image recognition processing described above may be performed on-board. Some of the processing may be performed off-board. A wireless connection can be established with another portable electronic device or computer having a second processor using the antenna142. The processor111of the smart necklace100and the second processor of the portable electronic device or computer can operate in conjunction with one another to perform the optical character or image recognition processing.

After the optical character or image recognition process is completed, and there is no document of interest for reading, the processor111may re-adjust the cameras to be in forward-facing positions. During optical character or image recognition, the stereo cameras121may be pointed downwards. The processor111recognizes using IMU data that the user1956stands up and starts walking. In response to this recognition, the processor111re-adjusts the field of view1990to be facing forward to focus on the surrounding environment and objects.

As used herein, the term “network” includes any cloud, cloud computing system or electronic communications system or method which incorporates hardware and/or software components. Communication among the parties may be accomplished through any suitable communication channels, such as, for example, a telephone network, an extranet, an intranet, Internet, point of interaction device, point of sale device, personal digital assistant (e.g., an Android device, iPhone®, Blackberry®), cellular phone, kiosk, etc.), online communications, satellite communications, off-line communications, wireless communications, transponder communications, local area network (LAN), wide area network (WAN), virtual private network (VPN), networked or linked devices, keyboard, mouse and/or any suitable communication or data input modality. Specific information related to the protocols, standards, and application software utilized in connection with the Internet is generally known to those skilled in the art and, as such, need not be detailed herein.

“Cloud” or “Cloud computing” includes a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing may include location-independent computing, whereby shared servers provide resources, software, and data to computers and other devices on demand.

The steps of a method or algorithm described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by the processor111, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium, such as the memory112, is coupled to the processor111such that the processor111can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor111. The processor111and the storage medium may reside in an Application Specific Integrated Circuit (ASIC).

The methods/systems may be described herein in terms of functional block components, screen shots, optional selections and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the methods/systems may employ various integrated circuit components, e.g., memory elements, processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the methods/systems may be implemented with any programming or scripting language such as, VPL, C, C++, C#, Java, JavaScript, VBScript, Macromedia Cold Fusion, COBOL, Microsoft Active Server Pages, assembly, PERL, PHP, awk, Python, Visual Basic, SQL Stored Procedures, PL/SQL, any UNIX shell script, and XML with the various algorithms being implemented with any combination of data structures, objects, processes, routines or other programming elements. Further, it should be noted that the methods/systems may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and the like.

As will be appreciated by one of ordinary skill in the art, the methods/systems may be embodied as a customization of an existing system, an add-on product, upgraded software, a stand-alone system, a distributed system, a method, a data processing system, a device for data processing, and/or a computer program product. Furthermore, the methods/systems may take the form of a computer program product on a non-transitory computer-readable storage medium having computer-readable program code means embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROM, optical storage devices, magnetic storage devices, and/or the like.