Patent ID: 12235348

DETAILED DESCRIPTION

FIGS.1through12, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably-arranged system or device.

An electronic device, according to embodiments of the present disclosure, can include a personal computer (such as a laptop, a desktop), a workstation, a server, a television, an appliance, and the like. In certain embodiments, an electronic device can be a portable electronic device such as a portable communication device (such as a smartphone or mobile phone), a laptop, a tablet, an electronic book reader (such as an e-reader), a personal digital assistants (PDAs), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a virtual reality headset, a portable game console, a camera, and a wearable device, among others. Additionally, the electronic device can be at least one of a part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or a measurement device. The electronic device is one or a combination of the above-listed devices. Additionally, the electronic device as disclosed herein is not limited to the above-listed devices and can include new electronic devices depending on the development of technology. It is noted that as used herein, the term “user” may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

Certain electronic devices include a graphical user interface (GUI) such as a display that allows a user to view information displayed on the display in order to interact with the electronic device. Electronic devices can also include a user input device, such as keyboard, a mouse, a touchpad, a camera, among others. The various types of input devices allow a user to interact with the electronic device. The input devices can be operably connected to the electronic device via a wired or wireless connection. Certain electronic devices can also include a combination of a user input device and a GUI, such as a touch screen. Touch screens allow a user to interact with the electronic device via touching the display screen itself.

An example input device is a remote control, which can be used to enables a user to wirelessly control another electronic device, such as a television. For example, when a user presses a button on a remote control, the remote control can send a signal, that includes instructions corresponding to the particular button that was pressed, to the television. The instructions instruct the television to perform the particular action corresponding to the pressed button.

Embodiments of the present disclosure recognize and take into consideration that input devices can be cumbersome to use. For example, if the remote control is misplaced, the user is unable to control the television. Additionally, many households include multiple televisions where each television can have a corresponding remote control with unique input button placements. As such, a user often memorizes the button placements on each remote for ease of controlling each television without needlessly inspecting the remote to perform simply functions such as changing a channel, adjusting the volume, controlling playback, and the like.

Accordingly, embodiments of the present disclosure provide user interface mechanisms and methods in which the user can interact with the electronic device while not necessarily touching the proprietary input device of an electronic device. For example, embodiments of the present disclosure provide systems and methods for controlling an electronic device based on a location of another device relative to the electronic device. For another example, embodiments of the present disclosure provide systems and methods for gesture recognition as an input for controlling an electronic device. A gesture refers to detected movements of an external object, over a period of time, that is used to control the electronic device. For example, a gesture can be the detected movement of a body part of the user, such as the hand or fingers of a user, or another electronic device, which is used to control the electronic device (without the user touching the device or an input device).

Embodiments of the present disclosure recognize and take into consideration that gestures can be used to control an electronic device, such as a television. However, gesture control, using a camera (such as a red-green-blue (RGB) camera or an RGB-depth (RGB-D) camera) can lead to privacy concerns, since the camera would effectively by monitoring the users constantly in order to identify a gesture. Additionally. camera-based gesture recognition solutions do not work well in all lighting condition, such as when there is insufficient ambient light.

Embodiments of the present disclosure provide user interface mechanisms and methods in which the user can interact with the electronic device, such as a television, based on presence and/or position detection & tracking of one or more other devices, such as a smart phone or wearable device. For example, if the electronic device (such as the television) and the other device (such as a wearable device) are both capable for wireless communication (such as using ultra-wide band (UWB) signals, BLUETOOTH, radar, WiFi, and the like), then the television can determine a relative location of the other device. The television can then determine an action to perform based on the relative location of the other device and/or gesture performed by the other device. Additionally, the television can determine an action to be performed on the other device based on the relative location of the other device and/or gesture performed by the other device.

In certain embodiments, the television can use UWB signals for determining the relative location of the other device. UWB signals enable an electronic device to obtain centimeter (cm) level ranging (the error in distance measurements between two electronic devices can be less than 10 cm in line-of-sight cases, and less than 50 cm in non-line-of-sight cases) and angle detection (with an error that is less than 3° in line-of-sight cases). It is noted that before ranging or position detection, the devices may already be discovered via a device discovery protocol that can be also UWB based or can be based on other wireless connectivity technology such as Bluetooth or Wi-Fi.

Embodiments of the present disclosure include systems and methods for determining, at a first electronic device (such as the television), a range and angle of arrival (AOA) measurement of a second electronic device (such as a phone) with respect to the first electronic device. The range and AOA measurement can be performed as a single instance or a combination of instances over a period of time. The combination of instances represents a gesture.

Embodiments of the present disclosure also include systems and methods for identifying and performing an action selected from a plurality of actions. The identified action can change the state on either the first electronic device (such as the television) or the second electronic device (such as a phone) based on the range and AOA measurement. In certain embodiments, the plurality of actions include performing authentication on the first electronic device, performing power savings on the first electronic device, initiating content playback, or controlling characteristics of content playback, and the like.

While the descriptions of the embodiments of the present discloser, describe UWB as the wireless technology for generating location information, similar technologies that be used (such as radar, WiFi, BLUETOOTH, and the like) to identify the relative location of external devices. That is, the embodiments of the present disclosure are not restricted to UWB and can be applied to other types of sensors that can provide both range and AOA measurements. It is noted that when applying the embodiments of the present disclosure using a different type of sensor (a sensor other than a measuring transceiver), various components may need to be tuned accordingly. Additionally, while this disclosure primarily uses a television as the primary electronic device (the first electronic device), and a phone or a watch as the external electronic device, it is noted that other device types can be applicable as well, such as a refrigerator with display panel, smart speaker, and the like.

FIG.1illustrates an example communication system100in accordance with an embodiment of this disclosure. The embodiment of the communication system100shown inFIG.1is for illustration only. Other embodiments of the communication system100can be used without departing from the scope of this disclosure.

The communication system100includes a network102that facilitates communication between various components in the communication system100. For example, the network102can communicate IP packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network102includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

In this example, the network102facilitates communications between a server104and various client devices106-116. The client devices106-116may be, for example, a smartphone, a tablet computer, a laptop, a personal computer, a wearable device, a head mounted display, or the like. The server104can represent one or more servers. Each server104includes any suitable computing or processing device that can provide computing services for one or more client devices, such as the client devices106-116. Each server104could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network102.

Each of the client devices106-116represent any suitable computing or processing device that interacts with at least one server (such as the server104) or other computing device(s) over the network102. The client devices106-116include a desktop computer106, a mobile telephone or mobile device108(such as a smartphone), a PDA110, a laptop computer112, a tablet computer114, and a television116. However, any other or additional client devices could be used in the communication system100, such as a wearable device or an appliance. Smartphones represent a class of mobile devices108that are handheld devices with mobile operating systems and integrated mobile broadband cellular network connections for voice, short message service (SMS), and Internet data communications.

In this example, some client devices108-116communicate indirectly with the network102. For example, the mobile device108and PDA110communicate via one or more base stations118, such as cellular base stations or eNodeBs (eNBs). Also, the laptop computer112, the tablet computer114, and the television116communicate via one or more wireless access points120, such as IEEE 802.11 wireless access points. Note that these are for illustration only and that each of the client devices106-116could communicate directly with the network102or indirectly with the network102via any suitable intermediate device(s) or network(s). In certain embodiments, any of the client devices106-116transmit information securely and efficiently to another device, such as, for example, the server104.

In certain embodiments, any of the client devices106-116can emit and receive UWB signals via a measuring transceiver. As illustrated, the television116and the mobile device108communicate via UWB signals. For example, the television116can transmit a UWB signal that is time stamped to the mobile device108. In response, the mobile device108transmits a time stamped UWB signal back to the television116. The television116can then generate location information, indicating a location of the mobile device108relative to itself (the television116). The location information can include distance between the television116and the mobile device108. Additionally, the location information can indicate angle features, such AOA measurements of the received UWB signals of the mobile device108. Based on the AOA measurements, the television116can determine an angle (in azimuth, elevation, or both) that the mobile device108is, relative to itself (the television116).

AlthoughFIG.1illustrates one example of a communication system100, various changes can be made toFIG.1. For example, the communication system100could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular configuration. WhileFIG.1illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIG.2illustrates an example electronic device in accordance with an embodiment of this disclosure. In particular,FIG.2illustrates an example electronic device200, and the electronic device200could represent the server104or one or more of the client devices106-116inFIG.1. The electronic device200can be a TV, such as the television116ofFIG.1. In other embodiments, the electronic device200can be a mobile communication device, such as, for example, a mobile station, a subscriber station, a wireless terminal, a desktop computer (similar to the desktop computer106ofFIG.1), a portable electronic device (similar to the mobile device108, the PDA110, the laptop computer112, or the tablet computer114ofFIG.1), a robot, and the like.

As shown inFIG.2, the electronic device200includes transceiver(s)210, transmit (TX) processing circuitry215, a microphone220, and receive (RX) processing circuitry225. The transceiver(s)210can include, for example, a RF transceiver, a BLUETOOTH transceiver, a WiFi transceiver, a ZIGBEE transceiver, an infrared transceiver, and various other wireless communication signals. The electronic device200also includes a speaker230, a processor240, an input/output (I/O) interface (IF)245, an input250, a display255, a memory260, and a sensor265. The memory260includes an operating system (OS)261, and one or more applications262.

The transceiver(s)210can include an antenna array including numerous antennas. The antennas of the antenna array can include a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate. As illustrated the transceiver210also includes a measuring transceiver270. The measuring transceiver270is discussed in greater detail below.

The transceiver(s)210transmits and receives a signal or power to or from the electronic device200. The transceiver(s)210receives an incoming signal transmitted from an access point (such as a base station, WiFi router, or BLUETOOTH device) or other device of the network102(such as a WiFi, BLUETOOTH, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The transceiver(s)210down-converts the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry225that generates a processed baseband signal by filtering, decoding, and/or digitizing the baseband or intermediate frequency signal. The RX processing circuitry225transmits the processed baseband signal to the speaker230(such as for voice data) or to the processor240for further processing (such as for web browsing data).

The TX processing circuitry215receives analog or digital voice data from the microphone220or other outgoing baseband data from the processor240. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry215encodes, multiplexes, and/or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The transceiver(s)210receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry215and up-converts the baseband or intermediate frequency signal to a signal that is transmitted.

The processor240can include one or more processors or other processing devices. The processor240can execute instructions that are stored in the memory260, such as the OS261in order to control the overall operation of the electronic device200. For example, the processor240could control the reception of forward channel signals and the transmission of reverse channel signals by the transceiver(s)210, the RX processing circuitry225, and the TX processing circuitry215in accordance with well-known principles. The processor240can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in certain embodiments, the processor240includes at least one microprocessor or microcontroller. Example types of processor240include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry. In certain embodiments, the processor240can include a neural network.

The processor240is also capable of executing other processes and programs resident in the memory260, such as operations that receive and store data. The processor240can move data into or out of the memory260as required by an executing process. In certain embodiments, the processor240is configured to execute the one or more applications262based on the OS261or in response to signals received from external source(s) or an operator. Example, applications262can include a multimedia player (such as a music player or a video player), a phone calling application, a virtual personal assistant, and the like.

The processor240is also coupled to the I/O interface245that provides the electronic device200with the ability to connect to other devices, such as client devices106-116. The I/O interface245is the communication path between these accessories and the processor240.

The processor240is also coupled to the input250and the display255. The operator of the electronic device200can use the input250to enter data or inputs into the electronic device200. The input250can be a keyboard, touchscreen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user in interact with the electronic device200. For example, the input250can include voice recognition processing, thereby allowing a user to input a voice command. In another example, the input250can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme, such as a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. The input250can be associated with the sensor(s)265, the measuring transceiver270, a camera, and the like, which provide additional inputs to the processor240. The input250can also include a control circuit. In the capacitive scheme, the input250can recognize touch or proximity.

The display255can be a liquid crystal display (LCD), light-emitting diode (LED) display, organic LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like. The display255can be a singular display screen or multiple display screens capable of creating a stereoscopic display. In certain embodiments, the display255is a heads-up display (HUD).

The memory260is coupled to the processor240. Part of the memory260could include a RAM, and another part of the memory260could include a Flash memory or other ROM. The memory260can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information). The memory260can contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc.

The electronic device200further includes one or more sensors265that can meter a physical quantity or detect an activation state of the electronic device200and convert metered or detected information into an electrical signal. For example, the sensor265can include one or more buttons for touch input, a camera, a gesture sensor, optical sensors, cameras, one or more inertial measurement units (IMUs), such as a gyroscope or gyro sensor, and an accelerometer. The sensor265can also include an air pressure sensor, a magnetic sensor or magnetometer, a grip sensor, a proximity sensor, an ambient light sensor, a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, a color sensor (such as a Red Green Blue (RGB) sensor), and the like. The sensor265can further include control circuits for controlling any of the sensors included therein. Any of these sensor(s)265may be located within the electronic device200or within a secondary device operably connected to the electronic device200.

In this embodiment, one of the one or more transceivers in the transceiver210is the measuring transceiver270. The measuring transceiver270is configured to transmit and receive signals for detecting and ranging purposes. The measuring transceiver270can transmit and receive signals for measuring range and angle of an external object relative to the electronic device200. The measuring transceiver270may be any type of transceiver including, but not limited to a WiFi transceiver, for example, an 802.1 lay transceiver, a UWB transceiver, and the like. In certain embodiments, the measuring transceiver270includes a sensor. For example, the measuring transceiver270can operate both measuring and communication signals concurrently. The measuring transceiver270includes one or more antenna arrays, or antenna pairs, that each includes a transmitter (or transmitter antenna) and a receiver (or receiver antenna). The measuring transceiver270can transmit signals at a various frequencies, such as in UWB. The measuring transceiver270can receive the signals from an external electronic device.

The transmitter, of the measuring transceiver270, can transmit UWB signals. The receiver, of the measuring transceiver, can receive UWB signals from other electronic devices. The processor240can analyze the time difference, based on the time stamps of transmitted and received signals, to measure the distance of the target objects from the electronic device200. Based on the time differences, the processor240can generate location information, indicating a distance that the external electronic device is from the electronic device200. In certain embodiments, the measuring transceiver270is a sensor that can detect range and AOA of another electronic device. For example, the measuring transceiver270can identify changes in azimuth and/or elevation of the other electronic device relative to the measuring transceiver270.

AlthoughFIG.2illustrates one example of electronic device200, various changes can be made toFIG.2. For example, various components inFIG.2can be combined, further subdivided, or omitted and additional components can be added according to particular needs. As a particular example, the processor240can be divided into multiple processors, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more neural networks, and the like. Also, whileFIG.2illustrates the electronic device200configured as a mobile telephone, tablet, or smartphone, the electronic device200can be configured to operate as other types of mobile or stationary devices.

FIG.3illustrates an example network configuration according to embodiments of the present disclosure. An embodiment of the network configuration shown inFIG.3is for illustration only. One or more of the components illustrated inFIG.3can be implemented in specialized circuitry configured to perform the noted functions or one or more of the components can be implemented by one or more processors executing instructions to perform the noted functions.

FIG.3illustrated a block diagram illustrating a network configuration including an electronic device301in a network environment300according to various embodiments. As illustrated inFIG.300, the electronic device301in the network environment300may communicate with an electronic device302via a first network398(e.g., a short-range wireless communication network), or an electronic device304or a server308via a second network399(e.g., a long-range wireless communication network). The first network398and/or the second network399can be similar to the network102ofFIG.1. The electronic devices301,302, and304can be similar to any of the client devices106-116ofFIG.1and include similar components to that of the electronic device200ofFIG.2. The server308can be similar to the server104ofFIG.1.

The electronic device301can be one of various types of electronic devices. The electronic devices may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the electronic devices are not limited to those described above.

According to an embodiment, the electronic device301may communicate with the electronic device304via the server308. According to an embodiment, the electronic device301may include a processor320, memory330, an input device350, a sound output device355, a display device360, an audio module370, a sensor module376, an interface377, a haptic module379, a camera module380, a power management module388, a battery389, a communication module390, a subscriber identification module (SIM)396, or an antenna module397. In some embodiments, at least one (e.g., the display device360or the camera module380) of the components may be omitted from the electronic device301, or one or more other components may be added in the electronic device301. In some embodiments, some of the components may be implemented as single integrated circuitry. For example, the sensor module376(e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented as embedded in the display device360(e.g., a display).

The processor320may execute, for example, software (e.g., a program340) to control at least one other component (e.g., a hardware or software component) of the electronic device301coupled with the processor320and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor320may load a command or data received from another component (e.g., the sensor module376or the communication module390) in volatile memory332, process the command or the data stored in the volatile memory332, and store resulting data in non-volatile memory334.

According to an embodiment, the processor320may include a main processor321(e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor323(e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor321. Additionally or alternatively, the auxiliary processor323may be adapted to consume less power than the main processor321, or to be specific to a specified function. The auxiliary processor323may be implemented as separate from, or as part of the main processor321.

The auxiliary processor323may control at least some of functions or states related to at least one component (e.g., the display device360, the sensor module376, or the communication module390) among the components of the electronic device301, instead of the main processor321while the main processor321is in an inactive (e.g., sleep) state, or together with the main processor321while the main processor321is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor323(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module380or the communication module390) functionally related to the auxiliary processor323.

The memory330may store various data used by at least one component (e.g., the processor320or the sensor module376) of the electronic device301. The various data may include, for example, software (e.g., the program340) and input data or output data for a command related thereto. The memory330may include the volatile memory332or the non-volatile memory334.

The program340may be stored in the memory330as software. The program340may include, for example, an operating system (OS)342, middleware344, or an application346.

The input device350may receive a command or data to be used by other components (e.g., the processor320) of the electronic device301, from the outside (e.g., a user) of the electronic device301. The input device350may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

In certain embodiments, the input device350can include a sensor for gesture recognition. For example, the input device350can include a transceiver similar to the measuring transceiver270ofFIG.2.

The sound output device355may output sound signals to the outside of the electronic device301. The sound output device355may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record, and the receiver may be used for incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.

The display device360may visually provide information to the outside (e.g., a user) of the electronic device301. The display device360may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, or projector. According to an embodiment, the display device360may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch. The display device360can be similar to the display255ofFIG.2.

The audio module370may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module370may obtain the sound via the input device350, output the sound via the sound output device355, or output the sound via a headphone of an external electronic device (e.g., an electronic device302) directly (e.g., wiredly) or wirelessly coupled with the electronic device301.

The sensor module376may detect an operational state (e.g., power or temperature) of the electronic device301or an environmental state (e.g., a state of a user) external to the electronic device301, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module376may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor. The sensor module376can be similar to the sensors265ofFIG.2.

The interface377may support one or more specified protocols to be used for the electronic device101to be coupled with the external electronic device (e.g., the electronic device302) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface377may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal378may include a connector via which the electronic device301may be physically connected with the external electronic device (e.g., the electronic device302). According to an embodiment, the connecting terminal378may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module379may convert an electrical signal into a mechanical stimulus (e.g., a vibration or a movement) or electrical stimulus which may be recognized by a user via his tactile sensation or kinesthetic sensation. According to an embodiment, the haptic module379may include, for example, a motor, a piezoelectric element, or an electric stimulator.

The camera module380may capture a still image or moving images. According to an embodiment, the camera module380may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module388may manage power supplied to the electronic device301. According to one embodiment, the power management module388may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery389may supply power to at least one component of the electronic device301. According to an embodiment, the battery389may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module390may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device301and the external electronic device (e.g., the electronic device302, the electronic device304, or the server308) and performing communication via the established communication channel. The communication module390may include one or more communication processors that are operable independently from the processor320(e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication.

According to an embodiment, the communication module390may include a wireless communication module392(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module394(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network398(e.g., a short-range communication network, such as BLUETOOTH, wireless-fidelity (Wi-Fi) direct, UWB, or infrared data association (IrDA)) or the second network399(e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module392may identify and authenticate the electronic device301in a communication network, such as the first network398or the second network399, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module396.

The antenna module397may transmit or receive a signal or power to or from the outside (e.g., the external electronic device) of the electronic device301. According to an embodiment, the antenna module397may include an antenna including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., PCB).

According to an embodiment, the antenna module397may include a plurality of antennas. In such a case, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network398or the second network399, may be selected, for example, by the communication module390(e.g., the wireless communication module392) from the plurality of antennas. The signal or the power may then be transmitted or received between the communication module390and the external electronic device via the selected at least one antenna.

According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module397.

At least some of the above-described components may be coupled mutually and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted or received between the electronic device301and the external electronic device304via the server308coupled with the second network399. Each of the electronic devices302and304may be a device of a same type as, or a different type, from the electronic device301. According to an embodiment, all or some of operations to be executed at the electronic device301may be executed at one or more of the external electronic devices302,304, or308. For example, if the electronic device301may perform a function or a service automatically, or in response to a request from a user or another device, the electronic device301, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service.

The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request and transfer an outcome of the performing to the electronic device301. The electronic device301may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology may be used, for example.

AlthoughFIG.3illustrates one example of the electronic device301in the network environment300, various changes can be made toFIG.3. For example, various components inFIG.3can be combined, further subdivided, or omitted and additional components can be added according to particular needs. As a particular example, the processor320can be further divided into additional processors, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more neural networks, and the like. Also, whileFIG.3illustrates the electronic device301configured as a mobile telephone, tablet, or smartphone, the electronic device301can be configured to operate as other types of mobile or stationary devices.

FIGS.4A and4Billustrate two way ranging (TWR) according to embodiments of the present disclosure. In particular,FIG.4Aillustrates an embodiment400of single sided two way ranging (SS-TWR) according to embodiments of the present disclosure.FIG.4Billustrates an embodiment450of double sided two way ranging (DS-TWR) according to embodiments of the present disclosure. That is, the TWR ofFIGS.4A and4Bdescribe how an electronic device (such as a first electronic device, similar to the television116ofFIG.1) calculates a distance between itself and another electronic device. Any one of the client device106-116ofFIG.1can include internal components that can perform single sided two way ranging, double sided two way ranging, or both single sided and double sided two way ranging.

Embodiments of the present disclosure provide systems and methods for using wireless technology to determine a location (such as distance or range feature) of another device relative to a primary or first device. An example of an outdoor localization technology is the Global Positioning System (GPS). Examples of wireless technologies that can be used in indoor localization include Wi-Fi (e.g. Wi-Fi based positioning system/WPS), Bluetooth, and UWB, and the like.

One localization technique is trilateration. Trilateration is a method of determining the location of a device (called tag) in space, using the ranges (distances) measurements between the tag and multiple spatially separated anchors with known locations. The ranges between the tag and the anchors can be determined using two way ranging (TWR) between the tag and each of the anchors. Based on the determined ranges between the tag and the multiple anchors the location of the tag is determined.

Range features are the range measurements between two devices equipped with a certain signaling capability such as by transmitting and receiving signals in UWB. Using UWB, the round-trip time of a signal sent between two devices are measured with the precision in the order of nanoseconds, then the range values between the two devices can be determined, since the speed is known. High ranging accuracy is obtained by using high precision in the time measurements. For example, ranging error can be less than 10 cm when the two devices are within line-of-sight. Other wireless signals can be used to obtain range values. For example, an electronic device can determine distance using received signal strength indication (RSSI) in WiFi. In this case, the range values can be determined through a mapping from a measurable quantity, such as signal strength. Embodiments of the present disclosure take into consideration that while such mapping (such as by using signal strength) can be a reasonable indicator of distance between devices (signal strength is smaller when the distance between two devices is greater than 10 meters than when this distance is smaller than 10 meters), it is difficult to have an exact mapping or distance (such as the ability to identify a distance between the two devices when the measured signal strength is 10 dB) because the signal strength is highly dependent on other aspects, such as hardware difference, multipaths in the environment, and the like.

The embodiment400ofFIG.4Aand the embodiment450ofFIG.4Billustrate two devices (device A and device B) exchanging signals using UWB for determining the distance between the devices.

As illustrated in the embodiment400ofFIG.4A, two devices, that of device A and device B, perform SS-TWR. It is noted that device A and device B can be any of the client devices106-116ofFIG.1and include any internal components of the electronic device200ofFIG.2or the electronic device301ofFIG.3. For example, device A can be a television116and device B can be the mobile device108, as illustrated inFIG.1. In certain embodiments, device A and device B (ofFIGS.4A and4B) can include a transceiver, such as the measuring transceiver270ofFIG.2, for transmitting and receiving messages.

SS-TWR is performed to measure the round trip delay of a single message from the initiator to the responder and a response sent back to the initiator. In particular, the device A initiates the exchange of signals and the device B responds to complete the exchange to perform a SS-TWR. Each device precisely timestamps the transmission and reception times of the message frames. Based on the timestamps, device A, device B, or both devices A and B can calculate times Troundand Treplyand then by using Equation (1), below, determine the resulting time-of-flight.
{circumflex over (T)}prop=½(Tround−Treply)  (1)

As illustrated in the embodiment450ofFIG.4B, two devices, that of device A and device B, perform DS-TWR. It is noted that device A and device B can be any of the client devices106-116ofFIG.1. For example, device A can be a television116and device B can be the mobile device108, as illustrated inFIG.1.

DS-TWR reduces the estimation error induced by clock drifts from long response delays of the SS-TWR. For example, the embodiment450ofFIG.4B, illustrates three messages are used for reducing the estimation error induced by clock drift from long response delays. Device A is the initiator to initialize the first round trip measurement, while device B as the responder, both responds to complete the first round trip measurement, and initializes the second round trip measurement. Each device precisely timestamps the transmission and reception times of the messages. Based on the time stamps a resultant time-of-flight estimate, Tprop, can be identified, such as by using Equation (2), below:

T^prop=(Tround⁢⁢1×Tround⁢⁢2-Treply⁢⁢1×Treply⁢⁢2)(Tround⁢⁢1+Tround⁢⁢2+Treply⁢⁢1+Treply⁢⁢2)(2)

A range (such as a distance) between the device A and device B, can be identified using the resultant time-of-flight estimate from SS-TWR (of the embodiment400ofFIG.4A) or DS-TWR (of the embodiment440ofFIG.4B). Equation (3), below, describes identifying the distance between the device A and device B. In Equation (3), R is the range estimate, {circumflex over (T)}propis the value identified in Equation (1) or Equation (2), and c is the speed of light.
R={circumflex over (T)}prop×c(3)

FIGS.5A-5Fdescribe the ability to use various antenna positions for enabling an electronic device to identify a relative angle between itself and another electronic device. In particular,FIG.5Aillustrates an embodiment500of an electronic device identifying AOA measurements of signals from an external electronic device according to embodiments of the present disclosure.FIGS.5B,5C, and5Dillustrate embodiments510,520, and530, respectively, depicting example antenna placements according to embodiments of the present disclosure.FIG.5Eillustrates a diagram540of an example position of an external electronic device relative to an electronic device in two-dimensions according to embodiments of the present disclosure.FIG.5Fillustrates a diagram550of an example position of an external electronic device relative to an electronic device in three-dimensions according to embodiments of the present disclosure. Any one of the client device106-116ofFIG.1and can include internal components for identifying angle features as described inFIGS.5A-5F. For example, a client device can identify angle features by using a transceiver, such as the measuring transceiver270ofFIG.2, with two or more antennas.

In addition to range (or distance), angle features are used to identify a location of an external electronic device. As described above inFIGS.4A and4B, range represents the distance an external device (such as the mobile device108ofFIG.1) is from a primary electronic device (such as the television116ofFIG.1).

Angle features, such as AOA, indicates a relative angle that the external device (such as the mobile device108ofFIG.1) is from a first electronic device (such as the television116ofFIG.1). AOA features are the angle-of-arrival measurements of the second device with respect to the first device, and available when the first device has multiple UWB antennas. For a pair of antennas, the phase difference of the signal coming to each antenna from the second device can be measured, and then be used to determine the AOA of the second device. In certain embodiments, an electronic device can be equipped with two pairs of antennas to measure both azimuth and elevation angles. In particular, one pair of antennas is placed in the horizontal direction to measure the angle of arrival in the horizontal plane, (denoted as the azimuth angle). The other pair of antennas is placed in the vertical direction to measure the angle of arrival in the vertical plane (denoted as the elevation angle).

As illustrated in the embodiment500ofFIG.5A, the phase difference among antennas is used to extract the AOA information from UWB measurements. For example, AOA is calculated based on the phase difference of between the two antennas, that of RX1and RX2. The distance between the two antennas is fixed and described by Equation (4), below. In Equation (4), λ is the wavelength and β is a normalization factor to link the distance d and wavelength λ. In certain embodiments,

d<λ2
and β<1. Thereafter, the AOA αlis identified using the measured phase difference between two antennas. The phase difference Δθland the AOA αlsatisfy the relationship as described in Equation (5), below.

d=β⁢λ2(4)Δ⁢θl=θl(2)-θl(1)=π⁢β⁢sin⁡(αl)(5)

FIG.5Billustrates antenna orientations of a transceiver512for identifying the elevation AOA. The transceiver512can be similar to the measuring transceiver271ofFIG.2. For example, to identify the elevation AOA, the phase difference of the two antennas (antenna514aand antenna514bofFIG.5B) are placed in the elevation direction, as illustrated.

FIG.5Cillustrates antenna orientations of a transceiver522for identifying the azimuth AOA. The transceiver522can be similar to the measuring transceiver271ofFIG.2. For example, to identify the azimuth AOA, the phase difference of the two antennas (antenna524aand antenna524bofFIG.5C) are placed in the azimuth direction, as illustrated.

FIG.5Dillustrates antenna orientations of a transceiver532for identifying the azimuth AOA and the elevation AOA. The transceiver532can be similar to the measuring transceiver271ofFIG.2. For example, to identify the elevation AOA, the phase difference of the two antennas (antenna534aand antenna534bofFIG.5D) are placed in the elevation direction and to identify the azimuth AOA, the phase difference of the two antennas (such as the antenna534band the antenna534cofFIG.5D) are placed in the azimuth direction, as illustrated.

As illustrated inFIG.5E, the television542, (such as the television116ofFIG.1) can determine location information of the electronic device544(such as the mobile device108ofFIG.1), which can be represented, in two dimensional (2D) coordinate system. For example, the location information of the electronic device544can be based on the range and the angle (R, θ) from the reference point television542to the reference point of the electronic device544. The range, R, is the distance between the television542and the electronic device544. The angle θ is defined as the angle between the vector from the television's542reference point to the electronic device544and its projection onto the vertical plane perpendicular to the TV's screen. For another example, the location information of the electronic device544can be represented by (x, y) in Cartesian coordinate system, where the television542is the reference location designated as (0, 0). Further, the tracking of the device with respect to the TV over a period of time is a gesture.

In certain embodiments, the first device, such as the television542can determine the relative location of the electronic device544. Based on the relative location of the electronic device544, the television542can determine whether the electronic device544is within or outside a designated area. Based on the determination, the television542can identify a particular action to perform. Similarly, the television542can determine whether the electronic device544performs a gesture and thereafter identify a particular action to perform. The action can be performed on the television542and/or the television542can notify the electronic device544to perform the identified action.

As illustrated inFIG.5F, the television552, (such as the television116ofFIG.1) can determine location information of the electronic device554(such as the mobile device108ofFIG.1), which can be represented, in three dimensional (3D) coordinate system. For example, the location information of the electronic device554can be represented by (R, θ, φ), where θ and φ are the azimuth and elevation angles, respectively. Here given {right arrow over (OT)} represents the vector from the television's reference point to the other device, the angle θ is defined as the angle between {right arrow over (OT)} and its projection onto the vertical plane perpendicular to the TV's screen, the angle φ is defined as the angle between {right arrow over (OT)} and its projection onto the horizontal plane perpendicular to the TV's screen. For another example, the location information of the electronic device554can be represented by (x, y, z) in Cartesian coordinate system. Further, the tracking of the device with respect to the TV over time can be recognized as gestures to control functions on an electronic device.

In certain embodiments, the first device, such as the television552can determine the relative location of the electronic device554. Based on the relative location of the electronic device554, the television552can determine whether the electronic device554is within or outside a designated area. Based on the determination, the television552can identify a particular action to perform. Similarly, the television552can determine whether the electronic device554performs a gesture and thereafter identify a particular action to perform. The action can be performed on the television552and/or the television552can notify the electronic device554to perform the identified action.

FIGS.6A,6B, and6Cillustrate example placements of the antennas according to embodiments of the present disclosure.

FIG.6Aillustrates the electronic device610a.FIG.6Billustrates the electronic device610b.FIG.6Cillustrates the electronic device610c. The electronic devices610a,610b, and610c(collectively electronic device610) can be any of the client devices106-116ofFIG.1and include any internal components of the electronic device200ofFIG.2or the electronic device301ofFIG.3. For example, electronic device610can be similar to the television116.

The electronic device610can include one or more measuring transceiver, such as the measuring transceiver270ofFIG.2, for transmitting and receiving messages. In certain embodiments, the one or more measuring transceiver of the electronic device610can be a UWB transceiver. The following embodiments are examples of the electronic devices610are not intended to limit the number of measuring transceivers or locations of the measuring transceivers.

As illustrated in the embodiment600aofFIG.6A, a single measuring transceiver612is positioned at the along the top perimeter of the electronic device610a. Similarly, as illustrated in the embodiment600bofFIG.6B, a single measuring transceiver622is positioned near the middle of the electronic device610b.

As illustrated in the embodiment600cofFIG.6C, multiple measuring transceivers (such as the measuring transceiver632a, measuring transceiver632b, measuring transceiver632c, and measuring transceiver632d) (collectively measuring transceivers632) are positioned at various locations within the electronic device610c. In certain embodiments, ranging can be performed between each of the measuring transceivers632(anchor) and an external electronic device, such as the mobile device108ofFIG.1. A location of the external electronic device can be identified based on the ranging between the multiple measuring transceivers632.

In certain embodiments, other signal processing techniques can be used to localize an electronic device in a 3D space, such as multilateration. Multilateration is used to determine a devices location based on measuring time of arrivals of the signals. For example, multilateration can be used to estimate the 2D (x, y) or 3D location (x, y, z) of the electronic device, based on the relative location of the measuring transceivers632. As illustrated, since there are two measuring transceiver in the horizontal direction (such as measuring transceiver632aand632b) and two measuring transceiver in the vertical direction (such as measuring transceiver632aand632c) the electronic device610ccan identify the 3D location of an external electronic device.

FIGS.7A,7B, and7Cillustrates examples for proximity based device control based on distance according to embodiments of the present disclosure. In particular,FIG.7Aillustrates an example method700for device control based on distance between the electronic devices according to embodiments of the present disclosure.FIG.7Billustrates an example diagram710indicating different locations of an external electronic device relative to the electronic device according to embodiments of the present disclosure.FIG.7Cillustrates an example method720for device control based on multiple distance thresholds between the electronic devices according to embodiments of the present disclosure.

The methods700and720are described as implemented by any one of the client device106-116ofFIG.1and can include internal components similar to that of electronic device200ofFIG.2and the electronic device301ofFIG.3. However, the method700as shown inFIG.7Aand the method720as shown inFIG.7Bcould be used with any other suitable electronic device and in any suitable system.

As illustrated in the method700ofFIG.7A, the diagram710ofFIG.7B, and the method720ofFIG.7Can electronic device, (such as the television116and the television712ofFIG.7B) determines an action to perform based on the location of another device (such as the mobile device108ofFIG.1and the mobile devices714aand714bofFIG.7B).

In step702, the electronic device (such as the television712ofFIG.7B) performs ranging with another an external electronic device (such as the mobile devices714aand714bofFIG.7B). For example, the electronic device and the external electronic device can include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for performing the ranging, as described above inFIGS.4A and4B. In certain embodiments, the measuring transceivers are UWB transceivers. For example, the UWB transceiver on an electronic device can perform ranging with another UWB transceiver on another device that is within the radio range. The result of ranging provides the distance (such as in in meter) between the two UWB transceivers.

In step704, the electronic device (such as the television712ofFIG.7B) determines whether the distance to the external electronic device (such as the mobile devices714aand714bofFIG.7B) is beyond a threshold R. Based on the determination the electronic device performs action A of step706or action B of step708.

Referring toFIG.7B, the electronic device is the television712and the external electronic device is the mobile device714aand714b. As illustrated in the diagram710ofFIG.7B, the mobile device714ais within the threshold R716from the television712while the mobile device714bis outside the threshold R716.

Referring toFIG.7A, the electronic device (such as the television712ofFIG.7B) will perform action A (step706) when the external electronic device is within the threshold R (such as illustrated by the mobile device714aofFIG.7B). Alternatively, the electronic device (such as the television712ofFIG.7B) will perform action B (step708) when the external electronic device is outside the threshold R (such as illustrated by the mobile device714bofFIG.7B). The determined action (either action A of step706or action B of step708) can include performing action or changing a state on the electronic device (such as the television712) and/or instructing the external electronic device to perform an action.

In certain embodiments, the actual action performed, based on the ranging result, can also depend on the current state of the electronic device such as the television712and/or the external electronic device.

The method720ofFIG.7Cillustrates using multiple ranging threshold to determine different actions. For example, there can be more one ranging result dependent actions, which provides more refined control. For example, R1can be 10 meter, R2can be 5 meter. Depending on whether the ranging result (distance between the electronic device and the external electronic device) is less than R1, greater than or equal to R1and less than R2, or greater than or equal to R2, the electronic device (such as the television712) can determine whether to perform Action A, Action B, or Action C, respectively, as illustrated inFIG.7C. It is noted that ranging thresholds R1and R2are adjustable and can also be calibrated or configured by users or devices.

For example, in step721a first electronic device (such as the television116ofFIG.1or the television712ofFIG.7B) performs ranging with a second electronic device (such as the mobile devices108ofFIG.1). For example, the first electronic device and the second electronic device can include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for performing the ranging, as described above inFIGS.4A and4B. In certain embodiments, the measuring transceivers are UWB transceivers. The first electronic device can generate location information of the second electronic device. The location information can include range, an elevation angle, an azimuth angle, or any combination thereof.

In step722, the first electronic device determines whether the identified distance (based on the generated location information) between the first electronic device and the second electronic device is less than a first ranging threshold. When the distance between the first electronic device and the second electronic device is less than the first ranging threshold, the first electronic device performs action A (step723). Alternatively, when the distance between the first electronic device and the second electronic device is greater than or equal to the first ranging threshold, the first electronic device proceeds to step724.

In step724, the first electronic device determines whether the identified distance (based on the generated location information) between the first electronic device and the second electronic device is between the first ranging threshold and a second ranging threshold. When the distance between the first electronic device and the second electronic device is between the first ranging threshold and a second ranging threshold, the first electronic device performs action B (step725). Alternatively, when the distance between the first electronic device and the second electronic device is smaller than or equal to the second ranging threshold, the first electronic device proceeds to step726.

In step726, the first electronic device determines whether the identified distance (based on the generated location information) between the first electronic device and the second electronic device is equal to or smaller than the second ranging threshold. When the distance between the first electronic device and the second electronic device is equal to or smaller than the second ranging threshold, the first electronic device performs action C (step727).

The determined action (either action A of step723, action B of step725, or action C of step727) can include performing action or changing a state on the first electronic device and/or instructing the second electronic device to perform an action. After performing Action A of step723, Action B of step725, or Action C of step727, the first electronic device can return to step721and perform an additional ranging with the second electronic device.

In certain embodiments, the actual action performed based on the ranging result can also depend on the current state of the first electronic device and/or the second electronic device. In certain embodiments, more or less ranging thresholds can be used for determining various actions to perform. Additionally, more or less actions can be performed. For example, the number of different actions that can be performed can be based on the number of ranging threshold

AlthoughFIGS.7A and7Cillustrates example methods, various changes may be made toFIGS.7A and7C. For example, while the methods700and720are shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

FIGS.8A,8B, and8Cillustrates an examples method for proximity based device control based on field of view according to embodiments of the present disclosure. In particular,FIG.8Aillustrates an example method800for device control based on field of view according to embodiments of the present disclosure.FIG.8Billustrates an example method810for identifying field of view (FOV) information according to embodiments of the present disclosure.FIG.8Cillustrates an example method820for device control based on multiple field of view regions according to embodiments of the present disclosure.

The methods800,810and820are described as implemented by any one of the client device106-116ofFIG.1and can include internal components similar to that of electronic device200ofFIG.2and the electronic device301ofFIG.3. However, the method800as shown inFIG.8A, the method810as shown inFIG.8B, and the method820as shown inFIG.8Ccould be used with any other suitable electronic device and in any suitable system.

As illustrated in the method800ofFIG.8Aand the method820ofFIG.8C, a first electronic device, (such as the television116) determines an action to perform based on whether a second device (such as the mobile device108ofFIG.1) is within a FOV of the first electronic device.

The first electronic device generates location information based on obtained ranging and angle of arrival (AOA) measurement results with respect to the second electronic device (such as via the UWB signaling between the two electronic devices). The AOA can be two dimensional (either azimuth or elevation) or three dimensional (both azimuth and elevation). The range and the AOA can be used to determine whether the second electronic device is in the FOV of the first electronic device. FOV is a specific combination of a range of range values and a range of AOA values. For example, an FOV for a television can be defined as the region that has all the points having range values to the TV within 0-4 meters range, and AOA values measured from the TV within −50 to 50 degrees range (with AOA=0 degree corresponding to right in front of the TV). The result of in-FOV detection can be used to determine one or more actions on the first and/or the second electronic device. This is illustrated inFIGS.8A and8C. The actual action performed can depend on the current state of the first electronic device and/or the state of the second electronic device.

As illustrated in the method800ofFIG.8A, the first electronic device (such as the television116ofFIG.1) performs ranging with a second electronic device (such as the mobile device108ofFIG.1) (step802). For example, the first electronic device and the second electronic device can include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for performing the ranging, as described above inFIGS.4A and4Band AOA, as described above inFIG.5A. In certain embodiments, the measuring transceivers are UWB transceivers. For example, the UWB transceiver on the first electronic device can generate location information representing a range and AOA with another UWB transceiver on the second electronic device. The location information can indicate the location of the second electronic device relative to the first electronic device.

In step804, the first electronic device determines whether the second electronic device is within a field of view of the first electronic device. Based on the determination the first electronic device performs action A of step806or action B of step808. For example, the first electronic device will perform action A (step806) when the second electronic device is within the FOV of the first electronic device. Alternatively, the first electronic device performs action B (step808) when the second electronic device is outside the FOV of the first electronic device. The determined action (either action A of step806or action B of step808) can include performing action or changing a state on the first electronic device and/or instructing the second electronic device to perform an action.

In certain embodiments, range and AOA measurements may not sufficiently reliably or accurately determine the in-FOV condition. In these cases, a post-processing can be performed, which takes one or more the UWB signals as inputs, including range, AOA, and features related to the channel impulse response (CIR) of the UWB signals. The post processing is used to determine whether the second electronic device is in-FOV of the first electronic device or out-FOV of the first electronic device. Examples of the features related to the CIR include the signal strength or SNR of the first detected signal path, the signal strength or SNR of the strongest detected signal path and the like. The post-processing is illustrated inFIG.8Band described below. Some examples of the post-processing are machine learning based algorithms such as neural network, support vector machine, and the like.

The method810ofFIG.8Bcan be performed by the first electronic device, such as the television116ofFIG.1, or transmitted to a server, such as the server104ofFIG.1or a combination of different devices.

In step812, the first electronic device generates location information. The location information can indicate the location of the second electronic device relative to the first electronic device. For example, the location information can include range (distance between the first electronic device and the second electronic device), AOA angle (in azimuth and/or elevation), and features related to the CIR of the signals.

In step814, the first electronic device performs a FOV classifier. The FOV classifier uses the location information and features related to the CIR of the signals to determine the location of the second electronic device relative to one or more FOV regions. In step816, the first electronic device identifies from the FOV information whether the second electronic device is within or outside a particular FOV region.

FIG.8Cis similar toFIG.7C, described above. For example, the method820ofFIG.8Cuses multiple FOV regions to determine an action to perform, while the method720ofFIG.7Cdescribes using multiple ranging threshold to determine an action to perform. The method820ofFIG.8Ccan be extended to actions that are dependent on the 2D/3D location with the FOV of the second electronic device, or 2D/3D FOV region of the second electronic device. An FOV region is a range of distance values and a range of the angle values.

In step821, a first electronic device ice (such as the television116ofFIG.1) generates location information associated with a second electronic device (such as the mobile devices108ofFIG.1). For example, the first electronic device can identify range and AOA features of the second electronic device. In certain embodiments, the first electronic device and the second electronic device include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for identifying the ranging and AOA features, as described above inFIGS.4A,4B, and5A. In certain embodiments, the measuring transceivers are UWB transceivers. The first electronic device can generate location information of the second electronic device. The location information can include range, an elevation angle, an azimuth angle, or any combination thereof.

In step822, the first electronic device determines whether the second electronic device is within a first FOV region with respect to the first electronic device. When the second electronic device is within the first FOV region with respect to the first electronic device, the first electronic device performs action A (step823). Alternatively, when the second electronic device is outside the first FOV region with respect to the first electronic device, the first electronic device proceeds to step824.

In step824the first electronic device determines whether the location information indicates that the second electronic device is within a second FOV region with respect to the first electronic device. When the second electronic device is within the second FOV region with respect to the first electronic device, the first electronic device performs action B (step825). Alternatively, when the second electronic device is outside the second FOV region with respect to the first electronic device, the first electronic device proceeds to step826.

In step826the first electronic device determines whether the location information indicates that the second electronic device is within a third FOV region with respect to the first electronic device. When the second electronic device is within the third FOV region with respect to the first electronic device, the first electronic device performs action C (step827).

The determined action (either action A of step823, action B of step825, or action C of step827) can include performing action or changing a state on the first electronic device and/or instructing the second electronic device to perform an action. After performing Action A of step823, Action B of step825, or Action C of step827, the first electronic device can return to step821and identify additional range or AOA features associated with one or more second electronic devices.

In certain embodiments, the actual action performed based on the ranging result can also depend on the current state of the first electronic device and/or the second electronic device. In certain embodiments, more or less FOV regions can be used for determining various actions to perform. Additionally, more or less actions can be performed. For example, the number of different actions that can be performed can be based on the number of FOV regions.

AlthoughFIGS.8A,8B, and8Cillustrates example methods, various changes may be made toFIGS.8A,8B, and8C. For example, while the methods800,810, and820are shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

RegardingFIGS.7A,7C,8A and8C, the various actions to be performed can vary based on the location of one or more second electronic devices, and a state of any of the electronic devices.

In certain embodiments, the first electronic device can be a television (such as the Television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. The television can be in an idle mode (power saving) state (Action A) where the screen can be turned off. When the second electronic device is within a range R of the television (based onFIG.7A or7B) or within a FOV region of the television (based onFIG.8A or8B), the television can change from Action A to Action B, where Action B is an active state. In the active state (Action B), the television can turn on and display a picture, a welcome message, or information that can be customized by the user.

Furthermore, the second electronic device can also perform action based on the range or the FOV region detection result. For example, the second electronic device can activate an application associated with the television such as a TV remote control application, when the second electronic device is within a range R of the television (or within a FOV region of the television). For example, the first electronic device can be further controlled (changing volume, changing a channel, playing a content, and the like) by identifying a gesture performed by the second electronic device.

In certain embodiments, additional features can be included with the first electronic device and/or the second electronic device to avoid unintentional triggers that cause an electronic device to perform a particular action (such as when the user inadvertently brings his phone (the second electronic device) within a distance of the TV (the first electronic device) for turning on the TV, but the user does not want the TV to turn on, at that particular time). For example, the user can set the action that is triggered based on a proximity (or range) to be disabled during certain hours of the day (e.g. work hours or late night). For another example, when the user does not deliberately disable the action, the second electronic device, when accidently triggered, can shows the option of disabling the feature to the user to decide. For another example, a configuration on the second electronic device could enable/disable range or FOV region detection when the second electronic device is locked or its screen is off, to avoid unintentional triggering of actions. Other techniques can be used to ensure the feature of action triggering based on proximity does not reduce user experience with devices.

Additional features can be added on the first electronic device to avoid unintentional triggers of actions. For example, the user can enable the auto login of his streaming account (such as NETFLIX) only during certain hours of the day (such as during a period of time for relaxing in the evening). In another example, when the user does not deliberately set the time duration for the automatic login to services, the user interface on the TV can directly shows the option of disabling the feature to the user to decide. Another user configuration could be enabling/disabling range or FOV region detection when the second electronic device is locked or its screen is off, to avoid unintentional service login. Other techniques can be used to ensure the feature of action triggering based on proximity does not reduce user experience with devices.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. If the second electronic device has not been moved for a preset period of time (such as 5 minutes), regardless of whether the second device is in FOV or not, the second electronic device can be put in a power saving mode with the range and AOA detection feature turned off.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. If the second electronic device has not been moved for a preset period of time (such as 5 minutes), regardless of whether the second electronic device is in FOV or not, the first electronic device can notify the second electronic device to enter a power saving mode and ignore when range and AOA detection from the first device. When the second electronic device start moving again, the first electronic device will set the first electronic device back to normal.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. If the second electronic device has not been moved for a preset period of time (for instance, can be but not limited to be 30 minutes, or finished a section of the show), regardless of whether the second electronic device is in FOV or not, the first electronic device can send a reminder such as “Turn off the device?” The first electronic device can be put into the power saving mode if there is no respond for within another preset period of time (for instance, can be but not limited to be 2 minutes).

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. If the second electronic device start the active state, the second electronic device can be set as device's user interface screen pop-up with the user interface of control tools, or keep the original device's user interface and let the control algorithm running in the back.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. In this embodiment, Action B is auto login of a service or account (such as a streaming service) run by the first electronic device, and Action A can be log off of the service or account. The service or account can be associated with the owner of the second electronic device that is in in proximity to the first electronic device, such that different service or account can be selected for the auto login depending on the device identity. The account authentication process takes into account the ranging result between the first electronic device and the second electronic device (such that authentication can pass when the ranging result which is securely sent to the authentication server determines if the device is within a required range for authentication).

For example, when the second electronic device is within a range R of the first electronic device (or within a FOV region of the first electronic device), the first electronic device can perform an auto login (Action B) of a service (such as a streaming service) or another account. Otherwise when the second electronic device is not within a range R of the first electronic device (or not within a FOV region of the first electronic device), the first electronic device can log out of the service or account (Action A). The service or account can be associated with the owner of the second electronic device. For example, the first electronic device can identify the first electronic device and be preprogramed to know that the second electronic device is associated with a particular user. Therefore, when the second electronic device associated with a particular user is within a certain range of the first electronic device, the first electronic device determines to use the log in information associated with the particular user of the second electronic device to use when logging into the account or service. For another example, the first electronic device can send a request to the second device for log in credentials associated with the user of the second device. The second device can then send the credentials automatically or request that the user provide the credentials which are then transmitted by the second device to the first electronic device.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. In this embodiment, Action B can be auto play of a content such as video/movie on the first electronic device when the device is determined to be within a range or within a FOV region, Action A can be auto pause of the content playing when the second electronic device is determined to be out of range. For example, when the second device is within a range of FOV of the first device, the first device can initiate Action B for playing content (such as a movie, music or show). Alternatively, when the second device is not within a range of FOV of the first device, the first device can initiate Action A and pause the content from being played.

In certain embodiments, the first electronic device can be a television (such as the television116ofFIG.1), and the second electronic device can be a smart phone (such as the mobile device108ofFIG.1) or a smart watch. In this embodiment, action A can cause content to play on the second electronic device when the second electronic device is out of a range (or out of a FOV region) of the first electronic device. Action B can cause the content playing on the second electronic device to switch to the first electronic device when the second electronic device is within a range (or within a FOV region) of the first electronic device. Before the auto switching of the content from the second electronic device to the television and vice versa, a message can be shown on the second electronic device, the first electronic device, or both the first and second electronic devices, to confirm the switching action. The auto switching behavior can be conditioned additionally on a pre-configuration by the user, so that the user's intention for the auto switching behavior is understood beforehand by the devices.

In certain embodiments, Action A can be content playing on the second electronic device when the second electronic device is out of a range and Action B can be auto switch of the content playing from the second electronic device to the television when the second electronic device is within a range.

It is evident from the above embodiments that determination of Action A and Action B may also depend on the state of the first electronic device, the second electronic device, or both the first and second electronic devices, in addition to the location of the second electronic device with respect to the first electronic device. The states of the second electronic device and/or the first electronic device include (i) whether a video is being played on the second electronic device or the first electronic device, (ii) whether the first electronic device is power saving or idle mode, (iii) which service is active on the second electronic device or the first electronic device, and (iv) the like.

FIGS.9A,9B,9C, and9Dillustrates an examples for proximity based device control for multiple devices according to embodiments of the present disclosure. In particular,FIG.9Aillustrates an example method900for performing an action based on location information of multiple external electronic devices according to embodiments of the present disclosure.FIG.9Billustrates a table910indicating an action to perform based on the proximity of the multiple external electronic devices according to embodiments of the present disclosure.FIG.9Cillustrates a diagram920where describing.FIG.9Dillustrates an example method930for performing an action based on location information of multiple external electronic devices according to embodiments of the present disclosure.

The methods900and930ofFIGS.9A and9D, are described as implemented by any one of the client device106-116ofFIG.1and can include internal components similar to that of electronic device200ofFIG.2and the electronic device301ofFIG.3. However, the methods900as shown inFIG.9Aand the method930as shown inFIG.9Dcould be used with any other suitable electronic device and in any suitable system.

In certain embodiments, one device equipped with UWB module can perform ranging with multiple devices equipped with UWB modules, by performing one/two-way ranging to each device. For example, a first electronic device equipped with UWB module can perform ranging with multiple external electronic devices equipped with UWB modules. In other embodiments, the devices can be identified by the devices' MAC addresses or other uniquely identifiable ID within the network. That is, the proximity based device control processes as described between a first electronic device and a multiple external electronic devices can be extended to between a television and multiple devices.

As illustrated in the method900ofFIG.9A, a first electronic device, (such as the television116ofFIG.1) in step902, generates location information of one or more external electronic devices (such as the mobile device108ofFIG.1) relative to the first electronic device based on a differences between the transmitted signals and the received signals. For example, the first electronic device generates location information of the one or more external electronic devices, such as two different mobile phones (such as the mobile devices108ofFIG.1).

For example, the location information can include ranging information, FOV, angles, or the like. The first electronic device and the one or more external electronic devices can include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for performing the ranging, as described above inFIGS.4A and4B. In certain embodiments, the measuring transceivers are UWB transceivers.

In step904, the first electronic device identifies the one or more external electronic devices from the location information. For example, the first electronic device can receive a device ID included in the received signals from the one or more external electronic devices. The device ID's can be used to identify each of the one or more external electronic devices. For example, the first electronic device can identify a user associated with each of the external electronic devices based on each device's respective device ID. In certain embodiments, the device ID's and their corresponding user is previously stored on the first electronic device. In certain embodiments, the external electronic devices can indicate to the first electronic device whether they belong to the same user, based on whether the external electronic devices have been securely paired by the user before (or two each other), or if the devices have been used to log into a user's account.

In this process the first electronic device can utilize UWB range and possibly 2D or 3D AOA measurements of each device and the corresponding device identity, to perform tasks as a function of the position of the two or more devices with respect to the first electronic device and the identities of the devices.

After identifying the location of the one or more external electronic devices and their corresponding owners, the first electronic device, in step906, either performs an action, notifies any of the external electronic devices to perform an action, or a combination thereof based on the location of the one or more external electronic devices relative to the first electronic device and the owners of each device.

For example, the table910ofFIG.9B, describes different actions that can be performed based on the location information of two electronic devices that are used by different users. As illustrated in the table910, the first electronic device can be a television (such as the television116ofFIG.1), and the external electronic devices (such as device1and device2) can be a smart phone (such as the mobile device108ofFIG.1). Although, the table910illustrates two devices (device1and device2), the table910is not intended to be limiting as additional external devices and actions can be included in the table.

As illustrated in the table910, according to the method900, the first electronic device (such as the television116ofFIG.1) can identify whether device1(such as the mobile device108ofFIG.1) and/or device2(such as the mobile device108ofFIG.1) are within a range from the first electronic device (“viewing zone”). Based on whether device1and device2are within a range of the first electronic device, and knowing that device1and device2are associated with different users, the first electronic device can determine and perform an action based. The determined action can include causing the first electronic device to change its state. Additionally, based on the determined action, the first electronic device can instruct device1and/or device2to perform an action.

For example, when both devices (device1and device2) are out of range (or out of a FOV region), the first electronic device (such as a television) can take Action A, which can be to turn off or to go into an idle state. When device1is within range (or within a FOV region) and device2is out of range (or out of a FOV region), the first electronic device can take Action B for device1, and the action can be customized or specific to device1, such as activating and displaying preferred service or content for the user of device1on the display screen of the first electronic device. Likewise, when device2is within range (or within a FOV region) and device1is out of range (or out of a FOV region), the first electronic device can take Action C for device2, and the action can be customized or specific to device2, such as activating and displaying preferred service or content for the user of device1on the display screen of the first electronic device. When both device1and device2are within range (or within a FOV region), the first electronic device can take Action D, such as activating and displaying commonly preferred service or content for the users of device1and device2.

In certain embodiments, any of the Actions, A, B, C, or D could cause content to be played on the device1or device2instead of the first electronic device. For example, device1is within range (or within a FOV region) and device2is out of range (or out of a FOV region), the first electronic device is performing Action B (displaying preferred service or content for the user of device1on the display screen of the television) and then device2is within the range (or FOV) of the first electronic device, the first electronic device could cause the preferred service or content for the user of device2on the display screen of the device2. Upon the first electronic device determining that the device1is out of range (or out of a FOV region), the content that is playing on display screen of the device2can be transferred to the first electronic device.

As described above in step904ofFIG.9A, the first electronic device identifies the external electronic devices from the location information. For example, if the first electronic device receives device IDs from the external electronic devices, the device IDs can indicate whether the external electronic devices belong to different users or the same users. In certain embodiments, the multiple external electronic devices that belong to the same user, such as when a user is carrying a smart phone as well as wearing a smart watch. Other device types include tablet, AR/VR glass, ear buds, other wearables, and the like.

As illustrated in the diagram920ofFIG.9C, user1's devices (device A and B) are within the “viewing zone” and user2's devices (device C and D) are out of the “viewing zone”. However, in general, it is also possible that different devices belonging to the same user are at different ranges or FOV regions, since a user may not always carry/wear all devices belonging to the user at the same time.

As illustrated in the method930ofFIG.9D, a first electronic device, (such as the television116ofFIG.1) in step932, determines the set of user devices belonging to a user. For example, the first electronic device identifies one or more the external electronic devices and then determines which of the identified devices belong to each user. For example, the first electronic device groups one or more of the identified devices to a particular user.

In step934, the first electronic device performs ranging or FOV detection with all the devices discovered. Actions based on the raging or the FOV detection can be customized for the user. In step936, the first electronic device determines whether at least one of the external electronic devices cause a change in the state of the first electronic device. For example, at least one of the external electronic devices cause a change in the state of the first electronic device when the device leaves the range or FOV. If none of the electronic devices cause a change in the state of the first electronic device, then the first electronic device repeats step934. Alternatively, when at least one of the electronic devices cause a change in the state of the first electronic device, the first electronic device performs an action.

In certain embodiments, the first electronic device performs an action in step938, based on which device causes the first electronic device to change its state. That is, the first electronic device infers the user's intention from the multiple devices belonging to the user, based on the priority associated with the external electronic devices. Certain external electronic devices can have higher priority if the device is considered to be a wearable device that is actively worn. For example, active wearables such as the AR glass, the smart watch, the ear buds, are more likely to be strongly correlated with the user's location, when they are worn on the body of the user. When any of the external electronic devices are detected to be on-body (such as via an on body detection mechanism on the wearable device), they can have high priority than a smart phone (such as the mobile device108) or another external device that is detected to not be on the body of the user. In one example, if both the AR glass (are detected as being worn) and the smart phone of a user are detected by the first electronic device, then the ranging result or FOV detection result of the AR glass has the higher priority to determine the corresponding action, than that of the smart phone. For instance, if the AR glasses leave the FOV while indicating that they are worn and the smart phone remains in the FOV, the first electronic device performs an action based on the location of the AR glasses (since they have a higher priority). The reason is that it is possible that the user walked away from the TV wearing the AR glasses and left his phone behind. Similarly, if the AR glasses are not worn and remain in the FOV while the phone leaves the FOV, then the first electronic device performs an action according to the phone leaving since the phone would have the higher priority since the AR glasses were not indicated as being worn.

In another embodiment, a first electronic device such as a television can perform ranging or FOV detection with all the devices it discovered. There can be multiple devices of the same type in the detection range and have been discovered by the first electronic device. For example, when multiple smart watches are in the detection range of first electronic device, user can set a special gesture (such as a circle that is drawn in the air) to determine the smart watch corresponding to the special gesture has the higher priority. If these multiple devices belong to same login user, the user can manually set the order of the priority.

In certain embodiments, a first electronic device such as a television can perform ranging or FOV detection with all the electronic devices discovered. There can be different types of devices in the detection range and have been discovered by the first electronic device. For example, when a smart watch, a pair of ear buds, a smart phone are in the detection range of the first electronic device, a user can set a special gesture (such as a circle that is drawn in the air) to determine the device corresponding to the special gesture has the higher priority. If these multiple devices belong to same login user, user can manually set the order of the priority.

In another embodiment, a first electronic device such as a television can perform ranging or FOV detection with multiple external electronic devices. There can be different types of devices in the detection range and have been discovered by the first electronic device. For example, when a smart watch, a pair of ear buds, a smart phone is in the detection range of the first electronic device, if these multiple electronic devices belong to different users, a user can manually set the order of the priority according to the user associated with each electronic device. For instance, a children's priority can manually set as lower than a main user.

In certain embodiments, if a first electronic device is a television, the television can split its screen into several smaller screens depending on the number of external electronic devices discovered within the FOV. The maximum number of split screens allowed can be preset in the first electronic device. If the number of the discovered external electronic devices exceeds the max number of split screens allowed by the first electronic device, only the high-priority devices can be allowed to use the split screen of the first electronic device. The first electronic device can respond to the action of each of the external electronic devices by displaying its response on its corresponding split screen. When one or several external electronic devices leave the FOV of the first electronic device, the first electronic device can close the split screen for the other device out of the FOV.

AlthoughFIGS.9A and9Dillustrates example methods, various changes may be made toFIGS.9A and9B. For example, while the method900is shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps. Similarly, while the method930is shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

FIGS.10A and10Billustrate example methods1000and1010for gesture based device control according to embodiments of the present disclosure. The methods1000and1010are described as implemented by any one of the client device106-116ofFIG.1and can include internal components similar to that of electronic device200ofFIG.2and the electronic device301ofFIG.3. However, the method1000as shown inFIG.10Aand the method1010as shown inFIG.10Bcould be used with any other suitable electronic device and in any suitable system.

The gestures described in the methods1000and1010are described in greater detail inFIGS.11A-11I.FIGS.11A-11Iillustrate example gestures according to embodiments of the present disclosure. In particular,FIG.11Aillustrates a push gesture1100a.FIG.11Billustrates a pull gesture1100b.FIG.11Cillustrates a push-pull gesture1100c.FIG.11Dillustrates a double tap gesture1100d.FIG.11Eillustrates a swipe right gesture1100e.FIG.1if illustrates a swipe left gesture1100f.FIG.11Gillustrates a wave gesture1100g.FIG.11Hillustrates a clockwise circle gesture1100handFIG.11Iillustrates a counter-clockwise circle gesture1100i.

In certain embodiments, when the first electronic device (such as a television) is equipped with an antenna array, the angle or direction of a second electronic device (external electronic device) such as a phone or a watch with respect to the first device can be detected and tracked by the first electronic device. In conjunction with the ranging result between the first electronic device and the second electronic device, a gesture that is performed by the second electronic device in front of the first electronic device can be determined by the first device. The identified gesture can be used to trigger an action on the first electronic device, the electronic second device, or both the first and second electronic devices. This process is illustrated inFIG.10A.

For a gesture based device control feature, it is possible to provide configuration options to user to avoid triggering unintentional actions. For example, there can be an option to enable/disable the gesture based device control feature when the external electronic device (such as a smart phone) is locked or when the screen of the external electronic device is off.

As describedFIGS.7A through9D, above, the first electronic device (such as a television) determines an action to perform, based on a particular location of an external electronic device. For example, based on whether an external electronic device is located within a particular range, or a particular FOV region (and possibly a state of one of the first electronic devices) with respect to the first electronic device (such as a television), the first electronic device performs a particular action. That is, the location information represents a single point in time. As described inFIGS.10A through11I, the location information represents a gesture that is based on location information that is generated over a period of time.

As illustrated in the method1000ofFIG.10A, a first electronic device generates location information of one or more electronic devices over a period of time (step1002). For example, the first electronic device, (such as the television116ofFIG.1) generates location information of one or more external electronic devices (such as the mobile device108ofFIG.1) relative to the electronic device based on a differences between the transmitted signals and the received signals over a period of time. The location information can be range and AOA. The location information can be expressed in 2D (such as inFIG.5E, described above) or in 3D (such as inFIG.5F, described above).

For example, the first electronic device and the external electronic device can both include measuring transceivers, (such as the measuring transceiver270ofFIG.2) for performing the ranging, as described above inFIGS.4A and4B. In certain embodiments, the measuring transceivers are UWB transceivers. For example, the UWB transceiver on a first electronic device can perform ranging with another UWB transceiver on another device that is within the radio range. The result of ranging provides the distance (such as in in meter) between the two UWB transceivers. Additionally, the UWB transceivers can include antenna such as described inFIGS.5B through5Dfor calculating the AOA. In certain embodiments, the UWB transceivers can be located in one location or multiple locations on the first electronic device as described inFIGS.6A through6C, above.

In step1004, the first electronic device receives the range measurements, angle measurements, or both range and angle measurements from the UWB transceiver. The first electronic device generates location information and performs gesture recognition. Based on the range, and AOA measurements over a period of time, the first electronic device identifies a gesture from the location information (range and/or angle).

For example, when the external electronic device is moved towards the first electronic device, the range between the two devices decreases. The first electronic device can identify the gesture as a push gesture as illustrated inFIG.11A. When the external electronic device is moved away from the first electronic device, the range between the two devices increases. The first electronic device can identify the gesture as a pull gesture as illustrated inFIG.11B.

When the external electronic device is moved away and then towards the first electronic device (or towards and then away from the first electronic device), the range between the two devices changes. The first electronic device can identify the gesture as a push-pull or pull push gesture as illustrated inFIG.11C. Similarly, when external electronic device is repeatedly moved away and then towards the first electronic device (or repeatedly moved towards and then away from the first electronic device), the range between the two devices changes repeatedly. The first electronic device can identify the gesture as a double tap gesture as illustrated inFIG.11D.

When the external electronic device is moved from a first location to the right, the angle between the two devices changes. The first electronic device can identify the gesture as a swipe right gesture as illustrated inFIG.11E. Similarly, when the external electronic device is moved from a first location to the left, the angle between the two devices changes. The first electronic device can identify the gesture as a swipe left gesture as illustrated inFIG.11F. When the external electronic device is moved from a first location and is waved to the left and right, the angle between the two devices changes repeatedly. The first electronic device can identify the gesture as a wave gesture as illustrated inFIG.11G.

When the external electronic device is moved in in a circle, with the distance between the first electronic device and the external electronic device not changing (or changing below a threshold), the azimuth angle and the elevation angle changes. Based on a pattern of when the azimuth angle and the elevation angle change, the first electronic device can identify the gesture as a clockwise circling gesture as illustrated inFIG.11Hor as a counter-clockwise circling gesture as illustrated inFIG.11I.

In step1006ofFIG.10A, the first electronic device performs an action according to the identified gesture. For example, the action could be changing the state of the first electronic device. For another example, the action could be notifying the external electronic device to perform an action. For yet another example, the action when performed causes the first electric device to change its state and notify the external electronic device to change its state, as well.

For example, the swipe right gesture ofFIG.11Eand the swipe left gesture ofFIG.11Fcan be used to move a pointer displayed on the first electronic device (such as the television116ofFIG.1), or to change content selection displayed on the first electronic device by swiping left or right.

For another example, the push-pull gesture ofFIG.11Ccan be used to perform selection of a content or item displayed on the first electronic device (such as the television116ofFIG.1). For another example, a push-pull gesture can result in a content that is currently being displayed or played on the external electronic device (such as a mobile device108ofFIG.1) to be displayed or played on the first electronic device. In certain embodiments, when the same gesture is performed again (or another gesture is performed) switches the display from the first electronic device to the external electronic device.

To detect and differentiate various gestures performed by the second device (external electronic device), the first electronic device uses the observed time varying range and AOA features.

FIG.10Billustrates the method1010for gesture detection and classification. In general, the method1010divides the gestures into two main categories. The first category corresponds to range gestures whose time-domain feature mainly varies in the detected range (such as illustrated in the gestures ofFIGS.11A-11D). The second category corresponds to angle gestures whose time-domain feature mainly varies in the detected AOA (such as illustrated in the gestures ofFIGS.11E-11I).

In step1012, the first electronic device (such as the television116ofFIG.1) generates location information corresponding to the external electronic device (such as the mobile device108ofFIG.1). For example, the first electronic device generates location information of one or more external electronic devices (such as the mobile device108ofFIG.1) relative to the first electronic device based on a differences between the transmitted signals and the received signals over a period of time.

In step1014, the first electronic device determines whether there is a significant enough motion present by checking that within a certain time T1, whether the variation of the range and the variation of AOA are larger than threshold R1and θ1. When the variation of the range is smaller than the threshold R1and the variation of AOA is smaller than the threshold θ1, the first electronic device determines that there was no motion and returns to step1012to generate more location information. Alternatively, when either range or AOA variation is larger than the threshold, the first electronic device determines that a valid motion is detected, and proceeds to step1016.

In step1016, the first electronic device observes the signal for a time duration of T2, where the start of the time duration T2can be the same of T1while the length of T2can be equal or larger than T1. After a valid motion of the external electronic device is detected, the first electronic device determines whether the location information corresponding to the external electronic device includes a variation in the range is larger than a threshold R2and the variation of AOA that is smaller than the threshold θ2.

When the external electronic device includes a variation in the range that is larger than the threshold R2and a variation of AOA that is smaller than the threshold θ2, then the first electronic device determines that the gesture is a range gesture and classifies the gesture in step1018. For example, based on how the range changes over a period of time, the first electronic device classifies the gesture as a push gesture as illustrated inFIG.11A, a pull gesture as illustrated inFIG.11B, a push-pull gesture as illustrated inFIG.11C, a double tap gesture as illustrated inFIG.11D, or another type of gesture.

When the first electronic device determines that the variation is in the range that is less than the threshold R2or a variation of AOA that is larger than the threshold θ2, then the first electronic device, in step1020further classifies the gesture. In step1020, the first electronic device determines whether the variation in the range that is less than or equal to the threshold R2and a variation of AOA that is larger than the threshold θ2. If both of the variation do not satisfy the condition (variation of the range is less than or equal to the threshold R2and the variation of AOA that is larger than the threshold θ2), then the first electronic device determines that there was an invalid motion detected and returns to step1012to generate more location information.

After determining that there was a valid motion, the first electronic device, in step1022inspects the particular elevation and azimuth variations of the AOA. That is, in step1022, the first electronic device determines whether the variation in the azimuth AOA are larger than or equal to the threshold θ3and whether the variation of elevation AOA are smaller than the threshold θ4. If both conditions are satisfied, then the first electronic device classifies the gesture as an azimuth angle gesture. For example, based on how the azimuth AOA changes over a period of time, the first electronic device classifies the gesture as a swipe right gesture as illustrated inFIG.11E, a swipe left gesture as illustrated inFIG.11F, a wave gesture as illustrated in FIGURE G, or another type of gesture.

If both conditions (of step1022) are not satisfied (such as when the variation of elevation AOA is larger than the threshold θ4) the first electronic device, in step1026determines whether the variation in the variation of azimuth AOA that is larger than or equal to the threshold θ3and whether the variation of elevation AOA that is larger than or equal to the threshold θ4.

If both conditions (of step1022) are not satisfied, then the first electronic device classifies the gesture was an invalid motion detected and returns to step1012to generate more location information.

Alternatively, if the first electronic device, in step1026determines that the variation of azimuth AOA that is larger than or equal to the threshold θ3and the variation of elevation AOA that is larger than or equal to the threshold θ4, then in step1028, the first electronic device classifies the gesture as a circle gesture. Additionally, based on how the elevation and azimuth change over a period of time, the first electronic device classifies the gesture as a clockwise circling gesture as illustrated inFIG.11H, a counter-clockwise circling gesture as illustrated inFIG.11I, or another type of gesture.

That is, the method1010describes that the first electronic device classifies the gestures to be range gestures and angle gestures based on whether the main time-domain feature variation is in the range or in the angle. Within the angle gestures, the first electronic device further divides the angle gestures based on whether the angle variation is mainly in azimuth direction or the variation is in both azimuth and elevation direction.

For example, the range gestures classification, as performed in step1018of theFIG.10B, the first electronic device identifies the end of the gesture. For example, the first electronic device can use a condition to indicate when the variation of the range is smaller than a threshold for a period of time. For example, during a time t, the variation of the range Var(R)<b. After finding the end of the gestures, the electronic device then use several conditions to differentiate between a pull gesture, a push gesture, a push-pull gesture, and a double tap gesture. The condition for a pull gesture is described in Equation (6), below. The condition for a push gesture is described in Equation (7), below. The condition for a push-pull gesture is described in Equation (8), below. The condition for a double tap gesture is described in Equation (9), below.
ΔR<−aANDt1<dANDt2<e(6)
ΔR>aANDt1<dANDt2<e(7)
ΔR1>fANDt1<dANDt2<eAND ∥R(P1)−R(P2)∥<g1  (8)
ΔR1>fANDt1<dANDt2<eAND ∥R(P1)−R(P2)∥<g1 AND ∥R(P3)−R(P2)∥<g1 AND ∥R(V1)−R(V2)∥<g1  (9)

The angle gestures classification as performed in step1020and1022of theFIG.10B, uses a transceiver with the antennas that are placed in the azimuth direction, the elevation direction, or both the azimuth and elevation directions, such as described inFIG.5B,5C, or5D. The azimuth angle can be represented by the expression A when classifying the angle gestures with azimuth angle variation (swipe left, swipe right, wave), as described in step1022. To identify the end of the gestures, the first electronic device uses a condition to indicate when the variation of the azimuth angle is smaller than a threshold for a period of time. For example, during a time t, the variation of the azimuth angle Var(A)<b. After finding the end of the gestures, the first electronic device then use several conditions to differentiate between a swipe left gesture, a swipe right gesture, or a wave gesture. The condition for a swipe left gesture is described in Equation (10), below. The condition for a swipe right gesture is described in Equation (11), below. The condition for a wave gesture is described in Equation (12), below.
ΔA>aANDt1<dANDt2<e(10)
ΔA<−aANDt1<dANDt2<e(11)
ΔA1>fANDt1<dANDt2<eAND ∥A(P1)−A(P2)∥<g1 AND ∥A(P3)−A(P2)∥<g1 AND ∥A(V1)−A(V2)∥<g1  (12)

If the first electronic device determines that the variation in both the azimuth angles and elevation angles is larger than certain thresholds, as shown in step1026ofFIG.10B, the electronic device then determines that the detected motion is a circling gesture. To classify the circling gesture as a clockwise circling gesture or a counter-clockwise circling gesture, the first electronic device inspects the time-series angle signal of both azimuth and elevation direction. For example, the electronic device segments the time-series elevation angle and the time-series azimuth angle into 3 types of regions. A region denoted by ‘S’ corresponds to a stable region. A region denoted by ‘+’ corresponds to a positive region. A region denoted by ‘−’ corresponds to a negative region. The regions are illustrated inFIGS.11H and11I.

An ‘S’ region is identified, during a period of time tn+1−tn, when the condition, as described in Equation (13) below, is satisfied. In Equation (13), var represents the variance. A region denoted by ‘+’ is identified, during a period of time tn+1−tn, when the condition, as described in Equation (14) below, is satisfied. A region denoted ‘−’ is identified, during a period of time tn+1−tn, when the condition, as described in Equation (15) below, is satisfied. In Equations (13), (14), and (15), c and b are a positive numbers. It is noted that to and tn+1are two observation time point. The time duration between these two observation time point is tn+1−tn.
∥var(tn,tn+1)∥<c(13)
A(tn+1)−A(tn)>b(14)
A(tn+1)−A(tn)<−b(15)

To differentiate between clockwise and counter-wise circling, the first electronic device identifies a pattern from the region variation sequence of the azimuth and elevation angles. For example, the clockwise circling gesture, the region variation sequence is shown inFIG.11H, while the counter-clockwise circling gesture, the region variation sequence is shown inFIG.11I.

For the clockwise circling gesture, the pattern can start with step1and continue with steps2-4and then back to step1. It is noted, the start of the clockwise circling gesture can also be step2,3, or4. If the clockwise circling gesture is performed with steps1,2,3,4,1,2, as illustrated inFIG.11H, then the time-series angle variation region sequence pair (azimuth angle region, elevation angle region) for clockwise circling can represent the following (S, −), (−, S), (S, +), (+, S), (S, −), (−, S).

For the counter-clockwise circling gesture, it can start with step1and continue with step2-4and then back to step1. It is noted, that the start of the counter-clockwise can also be step2,3, or4. If the counter-clockwise circling gesture is performed with steps1,2,3,4,1,2, as illustrated inFIG.11I, then the time-series region sequence pair (azimuth angle region, elevation angle region) for counter-clockwise circling can represent the following pattern (S, +), (−, S), (S, −), (+, S), (S, +), (−, S).

To differentiate clockwise and counter-clockwise circling, the electronic device uses the region sequence pairs of X number of steps, where X is equal or larger than two. For example, upon observing two consecutive region sequence pairs, such as (S, −) and (−, S), or (−, S) and (S, +), or (S, +) and (+, S), or (+, S) and (S, −), appear only in clockwise circling. Similarly, the two consecutive region sequence pairs (S, +) and (−, S), or (−, S) and (S, −), or (S, −) and (+, S), or (+, S) and (S, +) appear only in counter-clockwise circling.

The number of the circles the second electronic device draws can be determined by the number of region sequence pairs detected divide by four. For example, in the clockwise circling case, if six angle variation region pairs pattern (S, −), (−, S), (S, +), (+, S), (S, −), (−, S) are detected, then the total number of circles drawn is 6/4=1.5 circles.

In certain embodiments, the circling gesture can be used to increase or decrease the volume of the first electronic device depending on the circling directions. The number of circles that are drawn can be used to determine the numeric values that the volume is turned up or down.

In certain embodiments, the circling gesture can also be used to fast-forward or rewind the video playing on the first electronic device depending on the circling directions. The number of circles that are drawn can be used to determine the numeric value of the amount of the time that the video is fast-forwarding or rewinding.

In certain embodiments, one or more devices can be wearable devices that are worn by a user. For example, one device can be worn on a wrist of a user while another device is strapped to the ankle of the user. Using two wearable devices, enables the first electronic device to track the movements of the user and enable interactive gaming, as well as fitness training support.

In certain embodiments, volume control (such as increasing the volume or decreasing the volume) or content pause can be controlled when external electronic device is within viewing/hearing range as measured by first electronic device.

In certain embodiments, gesturing to first electronic device (such as a television116ofFIG.1) TV using the external electronic device (such as the mobile device108ofFIG.1) while a video call is on-going on the external electronic device causes the external electronic device to transfer the video call session to the first electronic device. For example, when the first electronic device identifies the gesture (as performed by the external electronic device), the first electronic device can identify the state of the first electronic device and the state of the external electronic device (the state here indicates the external electronic device is performing a video call). Thereafter, the first electronic device can generate a notification instructing the external electronic device to transfer the video call to the first electronic device.

In certain embodiments, when the external electronic device (such as the mobile device108ofFIG.1) is moved away from a viewing zone of the first electronic device (such as the television ofFIG.1), while video call is on-going on the first electronic device, the first electronic device can transfer the video call session from the first electronic device to the external electronic device.

In certain embodiments, because every house's size and layout is different, a user can set up the range regions and/or FOV regions according to the house layout. For example, a first electronic device (such as a television116ofFIG.1) can perform ranging or FOV detection with all the devices discovered. For example, the first electronic device is in a first room (such as a living room), but the user controls the first electronic device in another room (such as the kitchen). When the devices are in different rooms or areas of the house, the user can setup the range region and FOV region detection to be oriented toward the particular area or room (such as the kitchen).

AlthoughFIGS.10A and10Billustrates example methods, various changes may be made toFIGS.10A and10B. For example, while the method1000is shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps. Similarly, while the method1010is shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

FIG.12illustrates an example method1200for gesture recognition according to embodiments of the present disclosure. The method1200is described as implemented by any one of the client device106-116ofFIG.1and can include internal components similar to that of electronic device200ofFIG.2and the electronic device301ofFIG.3. However, the method1200as shown inFIG.12could be used with any other suitable electronic device and in any suitable system.

In step1202, the electronic device200transmits and receives signals. For example, the electronic device200transmits signals through a UWB transceiver (such as the measuring transceiver270ofFIG.2) After transmitting the signals, the electronic device can receive signals from an external electronic device.

In step1202, the electronic device200generates location information the external electronic device. The location information of the external electronic device is relative to the electronic device based on a differences between the transmitted signals and the received signals.

To generate the location information, the electronic device200identifies a time difference between a time-stamp included in the transmitted signals, and a time-stamp included in the received signals. Based on the time differences, the electronic device200determines a range (distance) between the electronic device200and the external electronic device. The electronic device200can also identify at least one AOA value of the received signals from the external electronic device, with respect to the electronic device.

Depending on the arrangement of the antenna in the transceiver (such as the measuring transceiver270) of the electronic device200, the electronic device200can determine an azimuth angle, an elevation angle, or both an azimuth and elevation angle between the electronic device200and the external electronic device. For example, when at least two of the antennas of the transceiver electronic device200are vertically positioned, the AOA value indicates an elevation angle between the electronic device200and the external electronic device. Similarly, when at least two of the antennas of the transceiver of the electronic device200are horizontally positioned, the AOA value indicates an azimuth angle between the electronic device200and the external electronic device.

It is noted that, the location information is two dimensional when the location information comprises the range and either (i) the azimuth angle or (ii) the elevation angle between the external electronic device and the electronic device. Similarly, the location information is three dimensional when the location information comprises (i) the range, (i) the azimuth angle, and (iii) the elevation angle between the external electronic device and the electronic device.

In certain embodiments, the electronic device200can determine location information using multilateration. For example, the electronic device200can includes multiple transceivers (such as the measuring transceiver270ofFIG.2). The electronic device200can generate the location information associated with an external electronic device by performing multilateration based on the transmitted and received signals from the multiple transceivers.

In certain embodiments, the location information can be based on a single instant in time. For example, the location information can indicate a location of the device at a particular time.

In certain embodiments, the location information can be based on multiple instances of time. For example, when the location information of the external electronic device is based on differences between the transmitted signals and the received signals over a period of time, the location information can represent a gesture or movement of the external electronic device. The electronic device200can identify a particular gesture based on particular changes in the ranges, the azimuth angle or the elevation angle

For example, the electronic device200can determine that the gesture is one of a ranging-type gesture based on (i) a comparison of distances between the electronic device and the external electronic device over the period of time to a ranging threshold, (ii) a comparison of angle values between the electronic device and the external electronic device over the period of time to an AOA threshold, and (iii) changes in the distances over the period of time. Based on the particular changes in the range, the electronic device200can identify whether the ranging-type gesture is a push gesture, a pull gesture, a push-pull gesture, a double tap gesture, or the like.

For another example, the electronic device200can determine that the gesture is one of an angle-type gesture based on (i) a comparison of the distances to the ranging threshold, a comparison of the angle values to the AOA threshold, and (ii) changes in the angle values over the period of time. Based on the particular changes in the range, the electronic device200can identify whether the angle-type gesture is a swipe left gesture, swipe right gesture, a wave gesture, a clockwise circle gesture, or a counterclockwise circle gesture, or the like.

In certain embodiments, to determine whether a gesture is a circular gesture, the electronic device200compares the angle values to an azimuth threshold, and the angle values to an elevation threshold. The electronic device then identifies a pattern based on changes in an azimuth angle and an elevation angle over the period of time. based on the pattern, the electronic device can determine whether the circular gesture is in a clockwise direction or in a counter-clockwise direction.

In step1208, the electronic device200determines an action that is to be performed based on the location information. In certain embodiments, if the electronic device200identifies a gesture from the location information, the determined action can be based on the identified gesture. In other embodiments, the electronic device can determine an action based on the actual location of the external electronic device. Additionally, the electronic device200can also use its own state (the state of the electronic device200), the state of the external electronic device, or both, when determining the action to be performed.

For example, the electronic device200can determine the action to be performed based on a comparison of the distance (as identified form the location information) between itself (the electronic device200) and the external electronic device to a threshold.

For another example, the electronic device can identify its current state, the current state of the external electronic device, or the state of both devices. The state can be whether a device is on, off, whether a device currently executing an application, and if so, the type of application being executed, whether a display of a device is on, whether content is playing or paused on the device, and the like. Based on the determined distance between the electronic device200and the external electronic device, and a state of one or both of the devices, the electronic device200can determine an action that is to be performed.

For yet another example, the electronic device200can determine an action to be performed based on whether the external electronic device is within a FOV of the electronic device200. For instance, the electronic device200can determine whether the external electronic device is within a FOV based on the location information. Based on whether the external electronic device is within or out of the FOV, the electronic device200can determine as action to be performed. When the external electronic device is within the FOV of the electronic device200, the electronic device200can determine to perform a first action. In contrast, when the external electronic device is not within the FOV of the electronic device200, the electronic device200can determine to perform a second action.

In certain embodiments, the electronic device200can generate location information associated with multiple external electronic devices. The multiple external electronic devices can be associated with a single user or two or more users. When generating the location information of multiple external electronic devices, the electronic device200can transmit and receive signals from the multiple external electronic devices. In some embodiments, the multiple external electronic devices can provide indication to the electronic device200. The identification can be used to identify the device and its particular owner. Some of the multiple electronic devices can also provide an indication as to whether the particular device is a wearable device and whether the external device is currently being worn by a respective user.

For example, when the electronic device200determines that the multiple external electronic devices are associated with a single user, the electronic device200can prioritize the external electronic devices. For instance, the electronic device200can prioritize a wearable device that is currently worn over a non-wearable device. Similarly, the electronic device200can prioritize a non-wearable device over a wearable device that is currently not worn. Based on the prioritization, the electronic device200determines an action for changing the state on at least one of (i) the electronic device200, or (ii) one of the external electronic devices is based on the additional location information.

For another example, when the electronic device200determines that the multiple external electronic devices are associated with different users, the electronic device200can determine an action to perform based on the location of the multiple external electronic devices. For example, when a first external electronic device is associated with a first person and a second external electronic device is associated with a second person, the electronic device200can determine if any of the external electronic devices are within a distance threshold relative to the electronic device200. The electronic device200determines an action to perform, based on which external device(s) is within the distance threshold.

In step1208, the electronic device200determines whether the determined action is associated with the electronic device200. If the determined action is associated with the electronic device200, then the electronic device200performs the action. The action could include changing a state of the electronic device200. For example, the action could be turning on, or turning off. For another example, the action could be logging into a service or account. For yet another example, the action could be changing a channel, changing the volume, playing content, pausing content, and the like.

In step1210, the electronic device200determines whether the determined action is associated with the external electronic device. If the determined action is associated with the external electronic device, then the electronic device200sends an instruction to the external electronic device. The instruction notifies the external electronic device to perform a particular action.

AlthoughFIG.12illustrates an example method, various changes may be made toFIG.12. For example, while the method1200is shown as a series of steps, various steps could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

The above flowcharts illustrate example methods that can be implemented in accordance with the principles of the present disclosure and various changes could be made to the methods illustrated in the flowcharts herein. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

Although the figures illustrate different examples of user equipment, various changes may be made to the figures. For example, the user equipment can include any number of each component in any suitable arrangement. In general, the figures do not limit the scope of this disclosure to any particular configuration(s). Moreover, while figures illustrate operational environments in which various user equipment features disclosed in this patent document can be used, these features can be used in any other suitable system.

Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims. None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claims scope. The scope of patented subject matter is defined by the claims.