Apparatus and method for sensing an environment

An apparatus includes a sensor for sensing an environment and a speaker. The apparatus also includes a display capable of displaying video information in multiple directions, wherein the display is configured to display video information in different directions based on information of an object sensed by the sensor. The speaker is configured to generate in multiple directions sound associated with the video information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims priority to Chinese Patent Application No. 201710539481.7, filed Jul. 4, 2017, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure generally relates to apparatus and method for sensing, outputting, and displaying. More particularly, the present disclosure relates to apparatus and method for apparatus with a human-machine interface for sensing an environment by a sensor and displaying video information in multiple directions by a display based on information of an object sensed by the sensor.

BACKGROUND

Individuals and businesses communicate in real time between various locations by means of video-teleconferencing devices. Such devices may be used in settings where people and/or content are in motion. However, utilizing current technology, it may be challenging to capture one or more callers while in motion. In some settings, a user may require a device that captures video in multiple directions or a 360° field of view. Such a requirement may be partly possible by using a 360° camera, for example. However, merely using a 360° camera has shortcomings. While a 360° camera may be able to generate a panoramic image, it cannot capture video information in different directions based on information of an object sensed by a camera.

SUMMARY

One aspect of the present disclosure is directed to an apparatus including a sensor for sensing an environment and a speaker. The apparatus also includes a display capable of displaying video information in multiple directions, wherein the display is configured to display video information in different directions based on information of an object sensed by the sensor, and the speaker is configured to generate in multiple directions sound associated with the video information.

Another aspect of the present disclosure is directed to a method which includes sensing an environment by a sensor and generating an output by a speaker. The method also includes displaying video information in multiple directions by a display, wherein the display displays video information in different directions based on information of an object sensed by the sensor, and the speaker generates in multiple directions sound associated with the video information.

Yet another aspect of the present disclosure is directed to a non-transitory computer readable medium storing instructions that are executable, the instructions being configured to cause a processor to sense an environment by a sensor and to generate an output by a speaker. The instructions are also configured to cause the processor to display video information in multiple directions by a display, wherein the display displays video information in different directions based on information of an object sensed by the sensor, and the speaker generates in multiple directions sound associated with the video information.

DETAILED DESCRIPTION

Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

Features and characteristics of the present disclosure, as well as methods of operation and functions of related elements of structure and the combination of parts and economies of manufacture, may become more apparent upon consideration of the following description with reference to the accompanying drawings, all of which form a part of this specification. It is to be understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. As used in the specification and in the claims, the singular form of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise.

The embodiments described herein relate to systems including one or more of image, video, and audio capture and display, including systems capable of consuming 360° content. Many situations, ranging from activities such as teleconferencing to recreational activity at panoramic sites, are ideal for capturing this type of content.

According to some embodiments of the disclosure, an apparatus includes a sensor to sense an environment, a speaker to generate an output, and a display to display video information in multiple directions, wherein the display is configured to display video information in different directions based on information of an object sensed by the sensor. Accordingly, the apparatus may be used by a user to teleconference, monitor, or view in real time one or more images captured by the apparatus. For example, in accordance with an embodiment of the disclosure, video data can be captured in multiple directions even if the user is in motion while the apparatus is capturing the video data.

According to some embodiments, the apparatus can be configured to capture one or more images in a variety of ways. For example, the apparatus can be configured to capture images, in response to user input, by software or hardware programming, by hardware setting, or by a combination thereof. In some embodiments, when the apparatus is configured to capture images by software or hardware programming or by hardware setting, image capturing can be performed subject to one or more predetermined conditions. For example, a set of predetermined conditions can trigger the apparatus to capture images. Alternatively, or additionally, the apparatus can capture images in response to a user's operation. In some embodiments, capturing images may include placing the apparatus in a particular operating mode or controlling a setting of the apparatus to enable a capability to capture one or more images. In some embodiments, capturing images may include capturing one or more images. As used herein, an “image” refers to, in part or in whole, a static or dynamic visual representation including, but not limited to, a photo, a picture, a graphic, a video, a hologram, a virtual reality image, an augmented reality image, other visual representations, or a combination thereof. In additional embodiments, the systems and methods disclosed herein further include a processor executing software instructions to identify an event based on a sound or image received by the apparatus.

As used herein, the term “real time” refers to the display of an image or video at the same time as, or within a short time of, its capture (on the order of milliseconds or tens of milliseconds), so that the displayed image or video realistically displays a typical moving object, such as a teleconference user or a baby crawling.

The embodiments described herein can apply to many fields. Descriptions and applications related to specific domains do not preclude the application of the described embodiments to other technologies or fields.

FIG. 1is an exemplary perspective view of an apparatus100consistent with embodiments of the present disclosure.FIG. 2is an exemplary top view of apparatus100. As seen inFIGS. 1 and 2, apparatus100has a square or rectangular cross section. Apparatus100includes a sensor for sensing an environment, a speaker, and a display capable of displaying video information in multiple directions. In some embodiments, the sensor includes one or more audio detectors, image detectors, a combination of different sensors, or any other sensing mechanism. For example, the sensor can include one or more of a microphone, a camera, an ultrasound sensor, a gyroscope, an accelerometer, or any other sensing mechanism capable of sensing an environment. In some embodiments, one or more of the sensors can be configured to sense a person in the environment. The one or more sensors can sense information of the sensed person including voice, location relative to apparatus100, commands, touches, and/or gestures of the person.

By way of example, apparatus100includes a housing102to hold a display104, a speaker array106, and sensors including a camera array108, and a microphone array110. Housing102has four lateral side surfaces102A-D and a top surface102E. Display104can include one display or multiple displays, for example displays104A-D, on respective side surfaces102A-D of housing102, of which displays104A and104B are directly visible inFIG. 1and displays104C and104D are on surfaces102C and102D of enclosure102not directly visible inFIG. 1. Camera array108includes cameras108A-D. In some embodiment, cameras108A-D may be configured to be mounted flush with lateral side surfaces102A-D of housing102, respectively. Microphone array110includes microphones110A-D. In some embodiments, microphones110A-D may be configured to be mounted on lateral side surfaces102A-D of housing102, respectively, of which microphone110A and110B are visible inFIG. 1. Speaker array106includes speakers106A-D. In some embodiments, speakers106A-D may be configured to be mounted on lateral side surfaces102A-D of housing102, respectively, of which speakers102A and102B are visible inFIG. 1.

In other embodiments, apparatus100is instead configured to have a cylindrical or spherical shape.

The display for displaying multiple images may wrap around the apparatus as do displays104A-D of apparatus100. In some embodiments consistent with the present disclosure, systems and methods may include more displays or as few as one display. By way of example, apparatus100may be placed at the center of a table. Users may sit on different sides of the table for a teleconference with a remote group of users. Each of the multiple displays may present the same image. The image may include the remote user that is speaking at that moment. Alternatively, each of the multiple displays may present a panoramic image including all remote users situated around remote apparatus. Further, the panoramic image received by the remote apparatus can split between each display of the apparatus. In another embodiment, the multiple images on the display can be still images or video of the remote group of users, or a user interface.

Cameras108A-D of camera array108can be configured to capture video data and have a multi-directional view of the environment. Video frames respectively generated by captured images may be combined and stitched to create a virtual reality experience. The term “virtual reality” as used herein refers to technology that uses images, sounds, and other sensations to simulate an environment, e.g., a real environment or an imaginary setting. In other embodiments consistent with the present disclosure, the systems and methods can include more cameras or as few as one camera.

Each of cameras108A-D may be a digital camera, a web camera, a fish-eye lens camera, or any camera with a multi-directional view of the environment. In some embodiments, cameras108A-D of camera array108may further be configured to capture a video with a native resolution of 320×240, 426×240, 640×360, 480×360, 800×600, 852×640, 960×540, 1024×768, 1280×720, 1280×960, 1280×1024, 1440×1050, 1440×1080, 1600×1200, 1920×1080, 2560×1440, 3840×2160, 5120×2880, 7680×4320, or any other suitable resolution. It will be understood that the forgoing examples are representative of exemplary embodiments only. The disclosed systems and methods may be implemented to provide enhanced resolution that is greater than the native or standard resolution capability.

In some embodiments, each of microphones110A-D of microphone array110can have directional pick-up fields corresponding to the fields of view of the cameras108A-D, respectively. In other embodiments consistent with the present disclosure, system and methods can include more microphones or as few as one microphone. In some embodiments, the microphone detects voice in the environment and the direction of the voice.

In some embodiments, speakers106A-D of speaker array106are multi-directional and each can be adjusted to play sound in different directions based on the information of the environment as sensed. In other embodiments consistent with the present disclosure, systems and methods can include more speakers or as few as one speaker.

By way of example, in some embodiments, apparatus100may generate local correlation information such that a remote user speaking may be displayed on display104while speakers106A-D output audio generated from the remote microphones corresponding to the remote camera(s) directed to the remote user. Moreover, apparatus100may be configured to only capture the image and voice of the person who is currently speaking.

In one embodiment, apparatus100is configured as a conferencing apparatus for use by multiple callers. In such an embodiment, display104displays information of a remote participant, at a remote site, in the direction of a first caller, using apparatus100at a local site. The information of the remote participant includes an image of the remote participant, the remote participant's location, or video/image that the remote participant shares with the first caller. The first caller at the local site may be moving around apparatus100. Apparatus100may be configured to capture a multi-directional view of the environment to follow the first caller's movement. In one embodiment, apparatus100can switch between cameras108A-D, for example, which have overlapping fields of view. As the first caller moves, the apparatus may capture image, audio, and correlation information of the first caller at any location at the local site. The apparatus can transmit continuous audio and video data by covering a multi-directional view of the environment at the local site in both vertical and horizontal directions around apparatus100. The display of an apparatus at the remote site may provide a continuous view of the moving first caller. The first caller does not need to adjust the physical orientation of apparatus100to transmit continuous video and audio data. Furthermore, the first caller that is moving at the local site may view or listen to the remote participant at the remote site without adjusting the local apparatus.

FIG. 3is an exemplary perspective view of a curved apparatus200consistent with embodiments of the present disclosure.FIG. 4is an exemplary top view of curved apparatus200. As seen inFIGS. 3 and 4, apparatus200has a cylindrical shape including a housing202having a circular cross section and a curved surface. Apparatus200includes a curved display204mounted on the curved surface of housing202. The curved display is suitable for displaying a wide-angle view or panoramic image, such as showing a virtual tour of a city or playing a movie. Apparatus200may be configured to recognize voice commands from a user and play a user-selected video or image on display204.

By way of example, apparatus200includes curved display204, a speaker array206, and sensors including a camera array208, and a microphone array210. Housing202has a curved surface202A and a top surface202B. Curved display204can include one display or multiple curved displays on curved surface202A of housing202. Camera array208includes cameras208A-D. In some embodiment, cameras208A-D may be configured to be mounted flush with curved surface202A of housing202, of which cameras208A and208B are directly visible and cameras208C and208D are on the portion of curved surface202A of enclosure202not directly visible inFIG. 3. Cameras208A-D of camera array208are directly visible in exemplary top view of curved apparatus200inFIG. 4. Microphone array210includes microphones210A-D. In some embodiments, microphones210A-D may be configured to be mounted on curved surface202A of housing202, of which microphone210A and210B are visible and microphones210C and210D are on the portion of curved surface202A of enclosure202not directly visible inFIG. 3. Speaker array206is provided as a continuous array of speakers around the circumference of curved surface202A of housing202, although speaker array206could instead be provided as, e.g., four discrete speakers distributed around housing202.

FIG. 5is an exemplary perspective view of a spherical apparatus300consistent with embodiments of the present disclosure. Apparatus300has a spherical shape including a housing302having a circular cross section and curved surface. The spherical shape of apparatus300allows for multi-directional use. In some embodiments, a display304is configured to display video information in different directions based on information of an object sensed by one or more sensors included therein, e.g., one or more cameras and microphones. In some embodiments, apparatus300includes a camera array306including cameras306A-F configured to provide a multi-directional view of the environment. Camera306D is not visible inFIG. 5. In some embodiments, apparatus300includes a microphone array308including microphones308A and308B configured to provide sound/voice detection in the local environment and a direction of the detected voice. Microphone array may include microphones on the side of apparatus300not visible inFIG. 5, to provide multi-directional sound/voice detection. In some embodiments, apparatus300includes a speaker array310configured to be multi-directional and adjustable to play sound in different directions based on information of the local environment as sensed. In one embodiment, the spherical apparatus may be hung from the ceiling to capture images in multiple directions. By way of example, apparatus300may be placed in a vault to enable surveillance of precious goods. Cameras306A-F may capture images in 360° vertical and horizontal directions of the vault. Upon entry by an intruder, speaker array310can generate an alarm output, display304can display real-time images of the intruder, and an associated processor can send a notification to emergency personnel.

Apparatus300can be configured to provide multi-directional detection of an environment. Since captured content can be multi-directional, audio and video data sensed by apparatus300can be viewed real-time by a remote user using a virtual reality device. Moreover, curved displays204ofFIG. 3 and 304can be configured to display virtual reality video data. Curved display204is suitable for displaying a wide-angle view or panoramic image. For example, curved display204can be configured to show a virtual tour of a city or play a movie. Apparatus100ofFIG. 1, 200ofFIG. 3, and 300are configured to capture, send, and receive multi-directional and virtual reality content.

FIG. 6is a block diagram of an exemplary system,500for controlling and coordinating operation of the plurality of elements of apparatus100ofFIGS. 1 and 2, apparatus200ofFIGS. 3 and 4, or apparatus300ofFIG. 5, according to some embodiments. System500includes a controller510, for example, an electronic device, capable of controlling apparatus100ofFIGS. 1 and 2to perform certain operations. In particular, controller510may be configured to control the timing of operations of display104, speaker array106, camera array108, and microphone array110of apparatus100, in order to synchronize operations of some or all of these components of apparatus100. In various embodiments, controller510may include a processor of local apparatus100, a remote apparatus, a remote control, a mobile phone, a tablet computer, a personal computer, a personal digital assistant (PDA), an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 player, etc. Controller510may be configured to form wireless and/or wired communications with each of the components of apparatus100. Controller510may include a user interface through which a user can enter various commands and data. For example, the user may use the user interface to control some or all of the components of apparatus100to perform one or more operations simultaneously. For example, all of the components of apparatus100may be controlled by controller510to monitor a room as a security and surveillance system or to operate as a family care system.

In exemplary embodiments, controller510may generate and transmit a plurality of wireless signals. In one embodiment, controller510may periodically transmit the plurality of signals at a predetermined time interval. In another embodiment, controller510may transmit the plurality of signals non-periodically.

In exemplary embodiments, controller510may transmit the wireless signals in any suitable manner. In one embodiment, controller510may transmit the wireless signals via a local network that connects both controller510and local apparatus100. In another embodiment, controller510may broadcast the wireless signals in certain broadcast channels, and apparatus100may scan for and receive the broadcast signals. In another embodiment, controller510may form a peer-to-peer (P2P) connection with apparatus100and transmit the wireless signals via the P2P connections.

In the context of this disclosure, synchronization does not necessarily mean all devices act at the same time; rather, the devices are synchronized if they act based on a common time reference or in a coordinated manner. Thus, in one aspect, apparatus100ofFIGS. 1 and 2receiving one wireless signal indicating a time delay may choose to activate one or more components at the time that is delayed from the current time by the amount of the time delay. Alternatively, apparatus100may choose to activate one or more components at another time based on the time delay indicated in the received wireless signal.

FIG. 7is a block diagram of an exemplary system400including apparatus100ofFIG. 1and a remote apparatus402, connected through a network404. As more fully described below, in system400processors execute instructions to receive and transmit audio and video data between local apparatus100and remote apparatus402over a wireless communication channel via network404.

Local apparatus100includes a processor406, a memory408, a communication port410, an input412, display104, camera array108, microphone array110, and speaker array106. Remote apparatus402may include corresponding elements, for example, processor414, memory416, communication port418, input420, display422, camera array424, microphone array426, speaker array428. In operation, processor406executes instructions (program code) and performs functions in accordance with techniques described herein. The instructions include routines, programs, objects, components, data structures, procedures, modules, and functions, which cause processor406to perform particular functions described herein. For example, processor406receives and analyzes video captured by cameras of camera array108. Processor406includes or is part of one or more known processing devices such as, for example, a microprocessor. In some embodiments, processor406includes any type of single or multi-core processor, mobile device microcontroller, central processing unit, etc. Processor406also controls display104to present video and/or images and speaker array106to emit audio data.

Memory408is configured to store one or more sets of instructions (program code) to be executed by processor406to perform exemplary methods disclosed herein. For example, memory408is configured to store instructions for execution by processor406, to send to and receive video and audio data from remote apparatus402. Remote apparatus402is any type of computing device. For example, remote apparatus402can be a smart phone, a tablet, a personal computer, a wearable device (e.g., Google Glass™ or smart watches, and/or affiliated components), or the like, or a combination thereof. In some embodiments, as for example illustrated inFIG. 7, remote apparatus402can include hardware and software elements corresponding to those of local apparatus100.

Memory408is also configured to store data and/or parameters used by processor406in methods described in this disclosure. For example, memory408stores one or more sound and visual models for detecting an event included in a video. Processor406can access the sound and video model(s) stored in memory408, and detect one or more events based on a sound or visual signal included in the video and the accessed sound or visual model(s) as described elsewhere herein. In some embodiments, memory408is configured to store one or more versions of video received from the cameras of camera array108.

Memory408can include one or more of memories that are volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other type of storage device or tangible (i.e., non-transitory) computer-readable medium including, but not limited to, a ROM, a flash memory, a dynamic RAM, and a static RAM.

Communication port410is configured to transmit data to and receive data from, among other devices, remote apparatus402over network404. Network404is any type of wired or wireless network that allows transmitting and receiving data. For example, network404is a wired network, a local wireless network, (e.g., Bluetooth™, WiFi, near field communications (NFC), etc.), a cellular network, the Internet, or the like, or a combination thereof. Other known communication methods which provide a medium for transmitting data between separate elements are also contemplated.

In some embodiments, cameras108A-D of camera array108are also configured to transmit captured video (or different versions thereof) to processor406, remote apparatus402, and/or any user device via network404. Additionally, processor406may also be configured to receive the video (or different versions thereof) from cameras108A-D of camera array108, remote apparatus402, and/or any user device via network404. Alternatively, cameras108A-D may be configured to transmit a video stream to processor406and remote apparatus402in real time. Additionally, processor406and remote apparatus402may be configured to receive a video stream from cameras108A-D in real time.

In some embodiments, camera array108, microphone array110, speaker array106, processor406, memory408, communication port410, and input412may be housed in a single device configured to perform functions of each element described in this disclosure. In some embodiments, processor406and memory408may be configured to perform one or more processes described in this disclosure. For example, camera array108may be configured to generate a video with a multidirectional, e.g., 360°, field of view and transmit the generated video to remote apparatus402, as described elsewhere in this disclosure.

In some embodiments, cameras of camera array108capture video data coupled with audio data captured by microphones of microphone array110. Processor406transmits the captured video and audio data to remote apparatus402wirelessly by network404. By way of example, display422and speaker array428of remote apparatus402are controlled by processor414to present captured video and audio data received from local apparatus100. In some embodiments, camera array424and microphone array426are controlled by processor414to capture video and audio data.

Local apparatus100is configured to receive real-time data (e.g., image and/or video data and audio data) from remote apparatus402via network404. Local apparatus100is also configured to present images and/or videos received from remote apparatus402to the local user via display104. Display104is any device configured to display, among other things, videos and/or images in display104based on the display data fed by processor406.

Input412is configured to receive inputs from remote apparatus402and transmit data/signal relating to the received inputs to processor406for further processing.

Processor406is configured to execute software instructions to receive local audio data from microphone array110and local video data from the camera array108. Processor406further generates local correlation information correlating local audio data from microphone array110with corresponding local video data from respective cameras of camera array108, based on the fields of view of camera array108and the pick-up fields of the microphone array110. Processor406also transmits local audio data, local video data, and local correlation information to remote apparatus402. In addition, processor406generates a plurality of images on display104from remote video data and generates outputs by speaker array106corresponding to the displayed images, based on remote audio data and remote correlation information received from remote apparatus402.

Memory416of remote apparatus402is configured to store one or more sets of instructions (program code) to be executed by processor414to perform exemplary methods disclosed herein. For example, memory416is configured to store instructions for execution by processor414, to send to and receive video and audio data from local apparatus100.

Communication port418of remote apparatus402is configured to transmit data to and receive data from, among other devices, local apparatus100over network404.

Input420of remote apparatus402is configured to receive inputs from local apparatus100and transmit data/signal relating to the received inputs to processor414for further processing.

FIG. 8is a flowchart of an exemplary process600for identifying a predetermined event and generating an alert, according to some embodiments. In some embodiments a sensor detects one or more objects in the environment and communicates information of the detected objects to a remote device. For example, the remote device can be a surveillance server, and the sensor can be included in apparatus100. As illustrated inFIG. 8, at602, microphones110A-D of microphone array110serve as sensors to sense acoustic signals and generate audio data. At604, processor406compares the generated audio data to sound models saved in memory408. At606, based on comparison between the generated audio data and the sound model, a predetermined sound event may be identified by processor406. At608, speakers106A-D generate an output alert. At610, processor406sends a notification. The notification may be sent to a specified telephone number, email address, or emergency personnel. In some embodiments, the system identifies a predetermined sound as a noise from a window shattering, an alarm system ringing, a baby crying, a person yelling for help, or any unusual sound in the room.

As illustrated inFIG. 8, at612, cameras108A-D serve as sensors to capture images and generate video data. At614, processor406compares video data to visual models saved in memory408. At616, based on comparison between the video data and the visual model, a predetermined visual event may be identified. At608, if the predetermined visual event has been identified, speakers106A-D generate an output alert. At610, processor406sends a notification. The notification may be sent to a specified telephone number, email address, or emergency personnel.

In some embodiments, cameras108A-D and microphones110A-D enable the system to provide a multidirectional view and sound detection, e.g., 360°, to follow a toddler's movement around a room and detect an accident or safety hazard. In some embodiments, the system may identify a predetermined visual event as an intruder entering, a fire, a person falling, or any unusual motion or foreign object in the room. In some embodiments, steps602through610(in identifying a predetermined sound event) and steps612through610(in identifying a predetermined visual event) are performed based on an exemplary process700shown inFIG. 9.

As illustrated inFIG. 9, at701, processor406receives video data, as described elsewhere in this disclosure. Processor406may detect one or more events based on video frames extracted from the video data. For example, in case of such detection, at702, processor406extracts a plurality of video frames from the video data. In some embodiments, processor406extracts the video frames from the video data continuously. Alternatively, one video frame is extracted within a period of time. By way of example, processor406may extract one video frame from every second or every minute of the video data. In some embodiments, the rate of extracting video frames is adjustable. For example, initially one video frame is extracted for every minute of the video data. An event may be detected at some time point of the video data (e.g., a moving object is detected). From that detection time point on (and/or a certain period of time before the time point), the rate of extracting video frames increases to, for example, 30 frames per minute from the previous rate of one frame per minute. The rate decreases if no more events are subsequently detected within a period of time. For example, the rate decreases back to one frame per minute if the moving object previously detected is not included in the video data within, for example, 10 minutes.

Processor406analyzes the extracted video frames at704. For example, processor406analyzes the video frames to identify an object included in the images. Processor406, at706, may detect one or more events based on analysis of the video frames. Exemplary events to be detected may include a motion event (e.g., a moving object is detected), an object recognition (e.g., a criminal suspect is recognized), event recognition (e.g., a baby stands up in crib), an emergency event (e.g., a fire incidence is detected), etc. For example, processor406detects a motion event included in video data by determining a difference in pixel values of a video frame and those of a preceding video frame. If the difference exceeds a threshold, a motion event is identified. Alternatively, as shown inFIG. 8, captured video data may be compared to visual models stored in the memory. Based on the comparison, a predetermined visual event may be identified.

At708, processor406determines whether any event is detected. If no event is detected (708—NO), at710, process700either ends or, alternatively, proceeds to701, and an event may be detected based on analysis of the data associated with the video (steps716through722), as described below. Additionally or alternatively, processor may determine whether any event is detected by simultaneously analyzing video data and audio data or by analyzing data in any order.

On the other hand, if one or more events are detected (708—YES), at712, processor406generates an alert, for example, an output from speakers106A-D. Processor406also identifies one or more video frames associated with the identified event. At714, processor406sends a notification to a specified telephone number, email address, or emergency personnel. In some embodiments, processor406may obtain information relating to the identified event(s) and/or information relating to the identified video frame(s) associated with the event(s). For example, processor406may obtain a time stamp (e.g., the starting time of the event) and/or a time window (e.g., the starting time and ending time of the event) for the detected event. Processor406may also obtain starting and ending points of the event. In some embodiments, processor406further identifies the video frames associated with the detected event (e.g., the video frames during the event, and within a period of time before and/or after the event). The obtained information may be communicated wirelessly to remote apparatus402via network404.

For example, in case of such detection, at716, processor406extracts a plurality of audio frames from the audio data. In some embodiments, processor406extracts the audio frames from the audio data continuously. Alternatively, one audio frame is extracted within a period of time. By way of example, processor406may extract one audio frame from every second or every minute of the audio data. In some embodiments, the rate of extracting audio frames is adjustable. For example, initially one audio frame is extracted for every minute of the audio data. An event may be detected at some time point of the audio data (e.g., a sound is detected). From that detection time point on (and/or a certain period of time before the time point), the rate of extracting audio frames increases to, for example, 30 frames per minute from the previous rate of one frame per minute. The rate decreases if no more events are subsequently detected within a period of time. For example, the rate decreases back to one frame per minute if the sound previously detected is not included in the audio data within, for example, 10 minutes. Processor406analyzes the extracted audio frames at718. For example, processor406analyzes the audio frames to identify a sound.

Processor406, at720, detects one or more events based on the analysis of the audio data. For example, processor406may detect a break-in event based on the detected sound of shattering glass (e.g., a window) in the audio data. At722, processor406determines whether there is any event detected. If no event is detected (722—NO), at724, process700either ends, or alternatively, proceeds to701, and an event can be detected based on analysis of video frames of the video data (steps702through708), as described above. Moreover, processor406may determine whether any event is detected by simultaneously analyzing video data and audio data or by analyzing data in any order. By way of example, processor406may determine whether there is any speech or any particular sound (e.g., laughter, crying, screaming, applause, glass shattering, etc.) included in the audio data.

On the other hand, if one or more events are detected (722—YES), processor406, at712, generates an alert. At714, processor406sends a notification. For example, the notification may be sent to a specified telephone number, email address, or emergency personnel. In some embodiments, a detected event based on the analysis of video frames is cross-referenced with the audio data of the video data to confirm the detected event, and vice versa.

FIG. 10is a flowchart of an exemplary process800for identifying events and displaying a product on a user interface. The term “product” as used herein refers to anything that can be offered for sale to a market that might satisfy a want or need, e.g. a gadget or clothing. As illustrated inFIG. 10, at802, microphones110A-D sense acoustic signals and generate audio data. At804, processor406compares the generated audio data to sound models saved in memory408. At806, based on comparison between the audio data and the sound model, a sound event may be identified. At808, display104displays an image based on the sound event identified in806. At810, display104displays a user interface.

As illustrated inFIG. 10, at812, cameras108A-D capture images and generate video data. At804, processor406compares generated video data to visual models saved in memory408. At814, based on comparison between the video data and the visual model, a visual event may be identified. At808, display104displays an image based on the visual event identified in814. At810, display104displays a user interface.

As further illustrated inFIG. 10, at816, cameras108A-D capture images of a one-dimensional or two-dimensional code and generate image data. The term “code” as used herein refers to any machine-readable optical label that contains information about an item to which it is attached, e.g. a matrix barcode, a Quick Response (QR) code. At804, processor406compares the generated image data to visual models saved in memory408. At818, based on comparison between the image data and the visual model, a code may be identified. At808, display104displays an image based on the code identified in818via network404. At810, display104displays a user interface.

In other embodiments, apparatus100detects one or more objects in the environment and communicates the information of the detected objects to a remote device. The remote device can be a vendor system that permits placement of an order for products based on the sensed information. In some embodiments, a sound event may consist of a command, for example, a command to search for a specified product. As another example, apparatus100recognizes a gesture performed by the user, the it may switch to an online shopping mode and receive user input regarding the shopping needs. In some embodiments, apparatus100may identify a product from QR code and display the product on display104via network404. The system may display a user interface to purchase the product or search similar products.

In some embodiments, cameras108A-D may be configured to capture an image of a real product and processor406identifies the product. Display104may present a user interface for the user to reorder the product or automatically search for online vendors of the same product or present search results for similar products.