Patent Description:
Mixed reality systems are a growing market. The tracking of inputs for traditional environments are often accomplished with a mouse device. The "Windows, Icon, Mouse, and Pointer" paradigm of desktop computing has remained largely unchanged since the late <NUM>. In the related art, <CIT> discloses a sensor based pointing device, <CIT> discloses an information processing apparatus, <CIT> discloses three dimensional pointing, and <CIT> discloses an input peripheral for a computer with 2D and 3D operation modes.

The advent of mobile computing, such as smartphones, tablets, and the like, has led to innovations in input and user experience such as multi-touch, styluses, and so forth. Similarly, today's Virtual and Augmented Reality (VR/AR) computing interfaces require specific remotes and controllers for user input, which are distinct from traditional computing mice.

This disclosure provides a system and method for a multipurpose input device for two-dimensional (2D) and three-dimensional (3D) environments.

In a first embodiment, an input device is provided. The device includes a position sensor capable of sensing two-dimensional (2D) position and a three-dimensional (3D) position, and a processor. The processor is configured to enter a 3D input mode when the processor detects (e.g., via information from the position sensor) that the input device is in a first orientation, and enter a 2D input mode when the processor detects (e.g., via information from the position sensor) that the input device is in a second orientation. The 2D input mode is configured to provide 2D position data to a connected computer system, and wherein the 3D input mode is configured to provide 3D position data to the connected computer system.

In a second embodiment, a method is provided. The method includes entering a 3D input mode when an input device is in a first orientation. The method also includes entering a 2D input mode when the input device is in a second orientation. The 2D input mode is configured to provide 2D position data to a connected computer system, and wherein the 3D input mode is configured to provide 3D position data to the connected computer system.

Embodiments of the present disclosure enable easy switching between distinct 2D and 3D modes. For example, using the same hybrid input device (multi-mode input device), a user can launch a game using a 2D menu and commence playing the game with a 3D controller. Embodiments of the present disclosure also enable continuous operations flow between 2D and 3D environments. For example, using the same hybrid input device (multi-mode input device) a user can browse a traditional 2D website, and, using the multi-mode input device as a 2D computer mouse, pull a 2D image from the page, and then place the 2D image in a 3D environment using the multi-mode input device as a 3D controller. In the aforementioned example, the user is not required to release the image to change input devices; but instead can maintain active control of the image by merely switching the multi-mode input device from a 2D mode to a 3D mode, such as by lifting the multi-mode input device away from the desk surface.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:.

Before undertaking the below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "transmit,", "receive," and "communicate," as well as derivatives thereof, encompass both direct and indirect communication.

As used herein, the terms "have,", "may have,", "include,", "may include,", "can have," or "can include" a feature (e.g., a number, function, operation, or a component such as a part) indicate the existence of the feature and do not exclude the existence of other features.

As used herein, the terms "A or B,", "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B,", "at least one of A and B,", "at least one of A or B" may indicate all of (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B.

As used herein, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other regardless of the order or importance of the devices. For example, a first component may be denoted a second component, and vice versa without departing from the scope of the present disclosure.

It will be understood that when an element (e.g., a first element) is referred to as being (operatively or communicatively) "coupled with/to," or "connected with/to" another element (e.g., a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that when an element (e.g., a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (e.g., a second element), no other element (e.g., a third element) intervenes between the element and the other element.

As used herein, the terms "configured (or set) to" may be interchangeably used with the terms "suitable for,", "having the capacity to,", "designed to,", "adapted to,", "made to," or "capable of" depending on circumstances. The term "configured (or set) to" does not essentially mean "specifically designed in hardware to. Rather, the term "configured to" may mean that a device can perform an operation together with another device or parts.

For example, the term "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (e.g., a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) for performing the operations.

The terms as used herein are provided merely to describe some embodiments thereof, but not to limit the scope of other embodiments of the present disclosure. It is to be understood that the singular forms "a,", "an", and "the" include plural references unless the context clearly dictates otherwise. All terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the present disclosure belong. In some cases, the terms defined herein may be interpreted to exclude embodiments of the present disclosure.

For example, examples of the electronic device according to embodiments of the present disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a PDA (personal digital assistant), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (e.g., smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch).

According to embodiments of the present disclosure, the electronic device can be a smart home appliance. Examples of the smart home appliance can include at least one of a television, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washer, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™) , a gaming console (XBOX™, PLAYSTATION™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

According to certain embodiments of the present disclosure, examples of the electronic device can include at least one of various medical devices (e.g., diverse portable medical measuring devices (a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, an sailing electronic device (e.g., a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller's machines (ATMs), point of sales (POS) devices, or Internet of Things devices (e.g., a bulb, various sensors, an electric or gas meter, a sprinkler, a fire alarm, a thermostat, a street light, a toaster, fitness equipment, a hot water tank, a heater, or a boiler).

According to certain embodiments of the disclosure, the electronic device can be at least one of a part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (e.g., devices for measuring water, electricity, gas, or electromagnetic waves).

According to embodiments of the present disclosure, the electronic device is one or a combination of the above-listed devices. According to embodiments of the present disclosure, the electronic device is a flexible electronic device. The electronic device disclosed herein is not limited to the above-listed devices, and can include new electronic devices depending on the development of technology.

As used herein, the term "user" may denote a human or another device (e.g., an artificial intelligent electronic device) using the electronic device.

<FIG>, discussed below, and the various embodiments used to describe the principles of this disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged wireless communication system.

The traditional "Windows, Icon, Mouse, and Pointer" (WIMP) paradigm is configured to provide two-dimensional (2D) inputs for an electronic device. The advent of mobile computing has led to innovations in input and user experience such as multi-touch, styluses, and the like. Similarly, today's Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Cross/Extended Reality (XR) and/or other computing interfaces require specific remotes and/or controllers for user input that are distinct from traditional computing mice. The existing VR/AR/MR/XR input approaches allow for three-dimensional (3D) manipulation of UI elements, but cannot emulate the precision input of a traditional computing mouse.

Embodiments of the present disclosure provide for a hybrid, multi-mode device that is able to function as both a computing mouse in a 2D environment as well as a VR/ AR/MR/XR controller for a 3D environment. The disclosed technology's hybrid system allows for computing experiences which seamlessly leverage both WIMP paradigms as well as new experiences enabled by VR/AR/MR/XR. Certain embodiments provide for VR/AR/MR/XR enhancements such as virtual representations that can be implemented with the device (e.g., virtual graphical icons displayed on the device to augment the device) to improve usability. Embodiments of the present disclosure can provide position/operation mode detection, ergonomics for seamless 2D and 3D gripping by user, 3D tracking for providing virtual representations of the physical device, user hand presence detection, user hand position sensing for providing virtual representation of user interaction, user hand position sensing for grip-dependent functionality and mode switching, 2D and 3D gesture support, as well as other features.

<FIG> illustrates an example electronic device in a network environment <NUM> according to various embodiments of the present disclosure. The embodiment of the electronic device <NUM> and the network environment <NUM> shown in <FIG> is for illustration only. Other embodiments of the electronic device <NUM> and network environment <NUM> could be used without departing from the scope of this disclosure.

According to an embodiment of the present disclosure, an electronic device <NUM> is included in a network environment <NUM>. The electronic device includes, or is coupled to, an input device <NUM>. The electronic device <NUM> can include at least one of a bus <NUM>, a processor <NUM>, a memory <NUM>, an input/output (IO) interface <NUM>, a display <NUM>, a communication interface <NUM>, or sensors <NUM>. In some embodiments, the electronic device <NUM> can exclude at least one of the components or can add another component.

The bus <NUM> includes a circuit for connecting the components <NUM> to <NUM> with one another and transferring communications (e.g., control messages and/or data) between the components.

The processor <NUM> includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> is able to perform control on at least one of the other components of the electronic device <NUM>, and/or perform an operation or data processing relating to communication.

The memory <NUM> can include a volatile and/or non-volatile memory. For example, the memory <NUM> can store commands or data related to at least one other component of the electronic device <NUM>. In various embodiments, the memory <NUM> can store spatial map data that can include mapping information of a real environment such as the interior of an office building, mall, house, amusement park, neighborhood or any other real world or virtual world mapping information utilized by an application <NUM> on the electronic device <NUM>. According to an embodiment of the present disclosure, the memory <NUM> stores software and/or a program <NUM>. The program <NUM> includes, e.g., a kernel <NUM>, middleware <NUM>, an application programming interface (API) <NUM>, and/or an application program (or "application") <NUM>. At least a portion of the kernel <NUM>, middleware <NUM>, or API <NUM> can be denoted an operating system (OS).

For example, the kernel <NUM> can control or manage system resources (e.g., the bus <NUM>, processor <NUM>, or a memory <NUM>) used to perform operations or functions implemented in other programs (e.g., the middleware <NUM>, API <NUM>, or application program <NUM>). The kernel <NUM> provides an interface that allows the middleware <NUM>, the API <NUM>, or the application <NUM> to access the individual components of the electronic device <NUM> to control or manage the system resources.

The middleware <NUM> can function as a relay to allow the API <NUM> or the application <NUM> to communicate data with the kernel <NUM>, for example. A plurality of applications <NUM> can be provided. The middleware <NUM> is able to control work requests received from the applications <NUM>, e.g., by allocating the priority of using the system resources of the electronic device <NUM> (e.g., the bus <NUM>, the processor <NUM>, or the memory <NUM>) to at least one of the plurality of applications <NUM>.

The API <NUM> is an interface allowing the application <NUM> to control functions provided from the kernel <NUM> or the middleware <NUM>. For example, the API <NUM> includes at least one interface or function (e.g., a command) for filing control, window control, image processing, or text control.

The IO interface <NUM> serve as an interface that can, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device <NUM>. Further, the IO interface <NUM> can output commands or data received from other component(s) of the electronic device <NUM> to the user or the other external device.

The display <NUM> includes, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectrome-chanical systems (MEMS) display, or an electronic paper display. The display <NUM> is able to display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display <NUM> can include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface <NUM> is able to set up communication between the electronic device <NUM> and an external electronic device (e.g., a first electronic device <NUM>, a second external electronic device <NUM>, or a server <NUM>). For example, the communication interface <NUM> can be connected with the network <NUM> or <NUM> through wireless or wired communication to communicate with the external electronic device. The communication interface <NUM> can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as video feeds or video streams.

Electronic device <NUM> further includes one or more sensors <NUM> that can meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, sensor <NUM> can include one or more buttons for touch input, a camera, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a depth or distance sensor, a grip sensor, a proximity sensor, a color sensor (e.g., a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared sensor (IR) sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and the like. The sensor(s) <NUM> can further include a control circuit for controlling at least one of the sensors included therein. Any of these sensor(s) <NUM> can be located within the electronic device <NUM>. A camera sensor <NUM> can capture a plurality of frames for a single image to be combined by the processor <NUM>.

The input device <NUM> is configured to detect a user motion and provide a corresponding command signal to the electronic device <NUM>. For example, in response to a movement of the input device <NUM> by a user, the electronic device <NUM> communicates the command signal corresponding to the movement of the input device <NUM> to the electronic device <NUM>. Additionally, an engagement, such as a pressing, of a button on the input device <NUM> can trigger a communication of the command signal. In certain embodiments, the input device <NUM> includes a touch pad and, in response to interactions with the touch pad, the input device <NUM> communicates the command signal. In response to the command signal, the electronic device <NUM> performs one or more functions. In certain embodiments, the input device <NUM> is coupled to the electronic device <NUM> via a wireless connection, such as a near-field connection such as BLUETOOTH, ZIGBEE, and the like, a magnetic connection, or an optical communication. In certain embodiments, the input device <NUM> is coupled to the electronic device via a wired or wireline connection.

In certain embodiments, an external electronic device, such as a wearable device or an electronic device <NUM>-mountable wearable device (e.g., an optical head mounted display (HMD)) is provided. When the electronic device <NUM> is mounted in the HMD, the electronic device <NUM> detects the mounting in the HMD and operate in an augmented reality mode. In certain embodiments, the electronic device <NUM> detects the mounting in the HMD and operate in an augmented reality mode. When the electronic device <NUM> is mounted in the HMD, the electronic device <NUM> communicates with the HMD through the communication interface <NUM>. The electronic device <NUM> can be directly connected with the HMD to communicate with the HMD without involving a separate network.

The wireless communication is able to uses a wireless communication medium, such as at least one of, e.g., long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (<NUM>), mm-wave or <NUM> wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard <NUM> (RS-<NUM>), or plain old telephone service (POTS).

The network <NUM> includes at least one of communication networks. Examples of communication include a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The external electronic devices <NUM> and server <NUM> each can be a device of the same or a different type from the electronic device <NUM>. According to certain embodiments of the present disclosure, the server <NUM> includes a group of one or more servers. According to certain embodiments of the present disclosure, all or some of operations executed on the electronic device <NUM> can be executed on another or multiple other electronic devices (e.g., the electronic device <NUM> or server <NUM>). According to certain embodiments of the present disclosure, when the electronic device <NUM> should perform some function or service automatically or at a request, the electronic device <NUM>, instead of executing the function or service on its own or additionally, can request another device (e.g., electronic device <NUM> or server <NUM>) to perform at least some functions associated therewith. The other electronic device (e.g., electronic device <NUM> or server <NUM>) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device <NUM>. The electronic device <NUM> can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique can be used, for example.

Although <FIG> shows that the electronic device <NUM> includes the communication interface <NUM> to communicate with the external electronic device <NUM> or server <NUM> via the network <NUM>, the electronic device <NUM> can be independently operated without a separate communication function, according to an embodiment of the present disclosure.

The server <NUM> can support to drive the electronic device <NUM> by performing at least one of operations (or functions) implemented on the electronic device <NUM>. For example, the server <NUM> can include a processing module or processor that may support the processor <NUM> implemented in the electronic device <NUM>.

For example, the electronic device <NUM> can include an event processing module, such as within processor <NUM>. The event processing module can process at least part of information obtained from other elements (e.g., the processor <NUM>, the memory <NUM>, the input/output interface <NUM>, or the communication interface <NUM>) and can provide the same to the user in various manners. The server event processing module can include at least one of the components of the event processing module and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module.

For example, according to an embodiment of the present disclosure, the event processing module processes information related to an event, which is generated while the electronic device <NUM> is mounted in a wearable device (e.g., the electronic device <NUM>) to function as a display apparatus and to operate in the augmented reality mode, to fit the augmented reality mode and display the processed information. When the event generated while operating in the augmented reality mode is an event related to running an application, the event processing module can block the running of the application or process the application to operate as a background application or process. Additional information on the event processing module <NUM> may be provided through <FIG> described below.

The event processing module can be separate from the processor <NUM> or at least a portion of the event processing module can be included or implemented in the processor <NUM> or at least one other module, or the overall function of the event processing module can be included or implemented in the processor <NUM> shown or another processor. The event processing module can perform operations according to embodiments of the present disclosure in interoperation with at least one program <NUM> stored in the memory <NUM>.

<FIG> illustrates an example electronic device <NUM> according to various embodiments of the present disclosure. The embodiment of the electronic device <NUM> shown in <FIG> is for illustration only. Other embodiments of electronic device <NUM> could be used without departing from the scope of this disclosure. The electronic device <NUM> depicted in <FIG> can be configured the same as, or similar to, any of electronic devices <NUM>, <NUM>, or <NUM>.

<FIG> is a block diagram illustrating an example configuration of an electronic device according to an embodiment of the present disclosure. Referring to <FIG>, the electronic device <NUM> according to an embodiment of the present disclosure can be an electronic device <NUM> having at least one display. In the following description, the electronic device <NUM> can be a device primarily performing a display function or can denote a normal electronic device including at least one display. For example, the electronic device <NUM> can be an electronic device (e.g., a smartphone) having a touchscreen <NUM>.

According to certain embodiments, the electronic device <NUM> can include at least one of a touchscreen <NUM>, a controller <NUM>, a storage unit <NUM>, or a communication unit <NUM>. The touchscreen <NUM> can include a display panel <NUM> and/or a touch panel <NUM>. The controller <NUM> can include at least one of an augmented reality mode processing unit <NUM>, an event determining unit <NUM>, an event information processing unit <NUM>, or an application controller <NUM>.

For example, when the electronic device <NUM> is mounted in a wearable device <NUM>, the electronic device <NUM> can operate, e.g., as an HMD, and run an augmented reality mode. Further, according to an embodiment of the present disclosure, even when the electronic device <NUM> is not mounted in the wearable device <NUM>, the electronic device <NUM> can run the augmented reality mode according to the user's settings or run an augmented reality mode related application. In the following embodiment, although the electronic device <NUM> is set to be mounted in the wearable device <NUM> to run the augmented reality mode, embodiments of the present disclosure are not limited thereto.

According to certain embodiments, when the electronic device <NUM> operates in the augmented reality mode (e.g., the electronic device <NUM> is mounted in the wearable device <NUM> to operate in a head mounted theater (HMT) mode), two screens corresponding to the user's eyes (left and right eye) can be displayed through the display panel <NUM>.

According to certain embodiments, when the electronic device <NUM> is operated in the augmented reality mode, the controller <NUM> can control the processing of information related to an event generated while operating in the augmented reality mode to fit in the augmented reality mode and display the processed information. According to certain embodiments, when the event generated while operating in the augmented reality mode is an event related to running an application, the controller <NUM> can block the running of the application or process the application to operate as a background process or application.

More specifically, according to an embodiment of the present disclosure, the controller <NUM> can include at least one of an augmented reality mode processing unit <NUM>, an event determining unit <NUM>, an event information processing unit <NUM>, or an application controller <NUM> to perform functions according to various embodiments of the present disclosure. An embodiment of the present disclosure can be implemented to perform various operations or functions as described below using at least one component of the electronic device <NUM> (e.g., the touchscreen <NUM>, controller <NUM>, or storage unit <NUM>).

According to certain embodiments, when the electronic device <NUM> is mounted in the wearable device <NUM> or the augmented reality mode is run according to the user's setting or as an augmented reality mode-related application runs, the augmented reality mode processing unit <NUM> can process various functions related to the operation of the augmented reality mode. The augmented reality mode processing unit <NUM> can load at least one augmented reality program <NUM> stored in the storage unit <NUM> to perform various functions.

The event detecting unit <NUM> determines or detects that an event is generated while operated in the augmented reality mode by the augmented reality mode processing unit <NUM>. Further, the event detecting unit <NUM> can determine whether there is information to be displayed on the display screen in relation with an event generated while operating in the augmented reality mode. Further, the event detecting unit <NUM> can determine that an application is to be run in relation with an event generated while operating in the augmented reality mode. Various embodiments of an application related to the type of event are described below.

The event information processing unit <NUM> can process the event-related information to be displayed on the display screen to fit the augmented reality mode when there is information to be displayed in relation with an event occurring while operating in the augmented reality mode depending on the result of determination by the event detecting unit <NUM>. Various methods for processing the event-related information can apply. For example, when a three-dimensional (3D) image is implemented in the augmented reality mode, the electronic device <NUM> converts the event-related information to fit the 3D image. For example, event-related information being displayed in two dimensions (2D) can be converted into left and right eye information corresponding to the 3D image, and the converted information can then be synthesized and displayed on the display screen of the augmented reality mode being currently run.

When it is determined by the event detecting unit <NUM> that there is an application to be run in relation with the event occurring while operating in the augmented reality mode, the application controller <NUM> performs control to block the running of the application related to the event. According to certain embodiments, when it is determined by the event detecting unit <NUM> that there is an application to be run in relation with the event occurring while operating in the augmented reality mode, the application controller <NUM> can perform control so that the application is run in the background so as not to influence the running or screen display of the application corresponding to the augmented reality mode when the event-related application runs.

The storage unit <NUM> can store an augmented reality program <NUM>. The augmented reality program <NUM> can be an application related to the augmented reality mode operation of the electronic device <NUM>. The storage unit <NUM> can also store the event-related information <NUM>. The event detecting unit <NUM> can reference the event-related information <NUM> stored in the storage unit <NUM> in order to determine whether the occurring event is to be displayed on the screen or to identify information on the application to be run in relation with the occurring event.

The wearable device <NUM> can be an electronic device including at least one function of the electronic device <NUM> shown in <FIG>, and the wearable device <NUM> can be a wearable stand to which the electronic device <NUM> can be mounted. In case the wearable device <NUM> is an electronic device, when the electronic device <NUM> is mounted on the wearable device <NUM>, various functions can be provided through the communication unit <NUM> of the electronic device <NUM>. For example, when the electronic device <NUM> is mounted on the wearable device <NUM>, the electronic device <NUM> can detect whether to be mounted on the wearable device <NUM> for communication with the wearable device <NUM> and can determine whether to operate in the augmented reality mode (or an HMT mode).

According to certain embodiments, upon failure to automatically determine whether the electronic device <NUM> is mounted when the communication unit <NUM> is mounted on the wearable device <NUM>, the user can apply various embodiments of the present disclosure by running the augmented reality program <NUM> or selecting the augmented reality mode (or, the HMT mode). According to an embodiment of the present disclosure, when the wearable device <NUM> functions with or as part the electronic device <NUM>, the wearable device can be implemented to automatically determine whether the electronic device <NUM> is mounted on the wearable device <NUM> and enable the running mode of the electronic device <NUM> to automatically switch to the augmented reality mode (or the HMT mode).

At least some functions of the controller <NUM> shown in <FIG> can be included in the event processing module <NUM> or processor <NUM> of the electronic device <NUM> shown in <FIG>. The touchscreen <NUM> or display panel <NUM> shown in <FIG> can correspond to the display <NUM> of <FIG>. The storage unit <NUM> shown in <FIG> can correspond to the memory <NUM> of <FIG>.

Although in <FIG> the touchscreen <NUM> includes the display panel <NUM> and the touch panel <NUM>, according to an embodiment of the present disclosure, the display panel <NUM> or the touch panel <NUM> may also be provided as a separate panel rather than being combined in a single touchscreen <NUM>. Further, according to an embodiment of the present disclosure, the electronic device <NUM> can include the display panel <NUM>, but exclude the touch panel <NUM>.

According to certain embodiments, the electronic device <NUM> can be denoted as a first device (or a first electronic device), and the wearable device <NUM> may be denoted as a second device (or a second electronic device) for ease of description.

According to certain embodiments, an electronic device can comprise a display unit displaying on a screen corresponding to an augmented reality mode and a controller performing control that detects an interrupt according to an occurrence of at least one event, that varies event-related information related to the event in a form corresponding to the augmented reality mode, and that displays the varied event-related information on the display screen that corresponds to the augmented reality mode.

According to certain embodiments, the event can include any one or more selected from among a call reception event, a message reception event, an alarm notification, a scheduler notification, a wireless fidelity (Wi-Fi) connection, a WiFi disconnection, a low battery notification, a data permission or use restriction notification, a no application response notification, or an abnormal application termination notification.

According to certain embodiments, the electronic device further comprises a storage unit configured for storing the event-related information when the event is not an event to be displayed in the augmented reality mode, wherein the controller can perform control to display the event-related information stored in the storage unit when the electronic device switches from the virtual reality mode into an augmented reality mode or a see-through (non-augmented reality) mode. According to certain embodiments, the electronic device can further comprise a storage unit that stores information regarding at least one event to be displayed in the augmented reality mode. According to certain embodiments, the event can include an instant message reception notification event. According to certain embodiments, when the event is an event related to running at least one application, the controller can perform control that blocks running of the application according to occurrence of the event. According to certain embodiments, the controller can perform control to run the blocked application when a screen mode of the electronic device switches from a virtual reality mode into an augmented reality mode or a see-through (non-augmented reality) mode. According to certain embodiments, when the event is an event related to running at least one application, the controller can perform control that enables the application, according to the occurrence of the event, to be run on a background of a screen of the augmented reality mode. According to certain embodiments, when the electronic device is connected with a wearable device, the controller can perform control to run the augmented reality mode. According to certain embodiments, the controller can enable the event-related information to be arranged and processed to be displayed in a three dimensional (3D) space of the augmented reality mode screen being displayed on a current display screen. According to certain embodiments, the electronic device <NUM> can include additional sensors such as one or more red, green, blue (RGB) cameras, dynamic vision sensor (DVS) cameras, <NUM> degree cameras, or a combination thereof.

<FIG> is a block diagram illustrating a program module according to an embodiment of the present disclosure. The embodiment illustrated in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. In the example shown in <FIG>, although an augmented reality (AR) system is depicted, at least some embodiments of the present disclosure apply equally to virtual reality (VR) and augmented reality (AR). Referring to <FIG>, the program module can include a system operating system (e.g., an OS) <NUM>, a framework <NUM>, and an application(s) <NUM>.

The system operating system <NUM> can include at least one system resource manager or at least one device driver. The system resource manager can perform, for example, control, allocation, or recovery of the system resources. The system resource manager may include at least one manager, such as a process manager, a memory manager, or a file system manager. The device driver may include at least one driver, such as, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a Wi-Fi driver, an audio driver, or an inter-process communication (IPC) driver.

According to certain embodiments, the framework <NUM> (e.g., middleware) can provide, for example, functions commonly required by an application or provide the application with various functions through an application programming interface (API) to allow the application to efficiently use limited system resources inside the electronic device.

The AR framework included in the framework <NUM> can control functions related to augmented reality mode operations on the electronic device. For example, when running an augmented reality mode operation, the AR framework <NUM> can control at least one AR application <NUM>, which is related to augmented reality, among applications <NUM> so as to provide the augmented reality mode on the electronic device.

The application(s) <NUM> can include a plurality of applications and can include at least one AR application <NUM> running in the augmented-reality mode and at least one normal application <NUM> running in a non-augmented-reality mode.

The application(s) <NUM> can further include an AR control application <NUM>. An operation of the at least one AR application <NUM> and/or at least one normal application <NUM> can be controlled by the AR control application <NUM>.

When at least one event occurs while the electronic device operates in the augmented reality mode, the system operating system <NUM> can notify the framework <NUM>, for example the AR framework, of an occurrence of an event.

The framework <NUM> can then control the running of the normal application <NUM> so that event-related information can be displayed on the screen for the event occurring in the non-augmented reality mode, but not in the augmented reality mode. When there is an application to be run in relation with the event occurring in the normal mode, the framework <NUM> can perform or provide control to run at least one normal application <NUM>.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM>, for example the AR framework, can block the operation of at least one normal application <NUM> to display the information related to the occurring event. The framework <NUM> can provide the event occurring, while operating in the augmented reality mode, to the AR control application <NUM>.

The AR control application <NUM> can process the information related to the event occurring while operating in the augmented reality mode to fit within the operation of the augmented reality mode. For example, a 2D, planar event-related information can be processed into 3D information.

The AR control application <NUM> can control at least one AR application <NUM> currently running and can perform control to synthesize the processed event-related information for display on the screen being run by the AR application <NUM> and display the result of the event related information thereon.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can perform control to block the running of at least one normal application <NUM> related to the occurring event.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can perform control to temporarily block the running of at least one normal application <NUM> related to the occurring event, and then when the augmented reality mode terminates, the framework <NUM> can perform control to run the blocked normal application <NUM>.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can control the running of at least one normal application <NUM> related to the occurring event so that the at least one normal application <NUM> related to the event operates in the background so as not to influence the screen used by the AR application <NUM> currently running.

Embodiments described in connection with <FIG> are examples for implementing an embodiment of the present disclosure in the form of a program, and embodiments of the present disclosure are not limited thereto and rather can be implemented in other various forms. Further, while the embodiment described in connection with <FIG> and <FIG> references AR, it can be applied to other scenarios such as mixed reality, or virtual reality etc. Collectively the various reality scenarios can be referenced herein as extended reality (XR).

<FIG> and <FIG> illustrate input devices according to this disclosure. The embodiments of the input devices 405a and 405b are for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

A 2D input device 405a, such as what is commonly referred to as a "mouse" is configured to rest on a tactile surface, such as a surface of a desk, table, or other suitable flat and substantially horizontal surface. The 2D input device 405a converts lateral motion (in two dimensions), such as movement right, left, upward (away from an operator), downward (towards the operator), or diagonally, into respective command signals for manipulating one or more objects, such as a cursor or icon, on a screen of the electronic device <NUM> to which the 2D input device 405a is coupled. In certain embodiments, the 2D input device 405a includes one or more buttons such as a first data button <NUM>, a second data button <NUM>, and a scroll wheel <NUM> to provide commands to the electronic device, such as a command to open or execute a particular application. The 2D input device 405a is configured to provide 2D inputs to the electronic device <NUM>. That is, the input device 405a is configured to detect motion in two dimensions and manipulate the cursor or icon in two dimensions.

A 3D input device 405b is configured to be held by an operator, such as away from a surface of a desk, table, or other suitable flat and substantially horizontal surface. The 3D input device 405b is able to track certain objects through at least three-degrees of freedom (3DOF). The 3D input device 405b includes sensors which track <NUM> or more degrees of freedom (DOF), such as 6DOF, typically orientation. The 3D input device 405b converts motion (in three dimensions), such as movement horizontally right, horizontally left, horizontally upward (away from an operator), horizontally downward (towards the operator), vertically upward (away from the surface or ground), vertically downward (towards the surface or ground), diagonally in any combination of the aforementioned directions, or rotationally, into respective command signals for manipulating one or more objects in a 3D environment on a screen of the electronic device <NUM> to which the 3D input device 405b is coupled. In certain embodiments, the 3D input device 405b includes a touch surface <NUM> and one or more buttons <NUM> to provide additional commands to the electronic device, such as a command to open or execute a particular application. The 3D input device 405b is configured to provide 3D inputs to the electronic device <NUM>. That is, the 3D input device 405b is configured to detect motion in three dimensions and manipulate an object in three dimensions.

In the age of spatial computing, users with computers running VR/AR (and/or MR/ XR) environments may require both computing mice, such as 2D input device 405a, as well as VR/AR controllers, such as 3D input device 405b. Such environments may often require users to frequently switch between input devices, or such otherwise environments may fall short when attempting to provide 2D precision using a 3D controller (or vice-versa).

VR also brings about unique challenges for the 2D input device 405a due to the inability for users to directly see the 2D input device 405a, such as while users are wearing VR headsets. This makes simple tasks, such as locating the 2D input device 405a or coordinating hand motions to buttons <NUM>, <NUM> on the 2D input device 405a, more difficult. Many of these challenges have been addressed for 3D input device 405b (VR controllers) but not for traditional input devices.

Embodiments of the present disclosure provide a hybrid, multi-mode input device that is able to function as both a 2D input device, such as a computing mouse, as well as a 3D input device, such as a VR/AR/MR/XR controller. The hybrid, multi-mode input device enables computing experiences that seamlessly leverage both WIMP paradigms as well as new experiences enabled by VR/AR/MR/XR.

Certain embodiments can reduce the number of required input devices, thus decreasing the need to switch devices, and also improving portability. Certain embodiments can enable new workflows that seamlessly transition between multiple modalities, such as dragging from 2D mouse to 3D workspace. Certain embodiments can further improve the user experience of computing mouse operations through VR representations of the device and finger/grip positioning.

<FIG> illustrate an example multi-mode input device according to embodiments of the present disclosure. The example embodiment of the multi-mode input device <NUM> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

The multi-mode input device <NUM> includes a housing <NUM> with an ergonomic design. The housing <NUM> can have a contoured portion that enables a user to comfortably place or rest a palm of their hand on the multi-mode input device <NUM>. The contoured portion is also configured to enable the user to grip the multi-mode input device <NUM> with their hand.

That is, the multi-mode input device <NUM> includes a 2D mouse configuration, as shown in <FIG>. The 2D mouse configuration can include <NUM> buttons and a scroll wheel, and is configured to provide 2D tracking. In operation of the 2D mouse configuration, a hand of the user is parallel to a tactile surface, such as a surface of a desk or table.

That is, the multi-mode input device <NUM> also includes a 3D controller configuration, as shown in <FIG>. The 3D controller configuration can include <NUM> triggers (switched function from buttons) and a trackpad, and is configured to provide 3D tracking. The multi-mode input device <NUM> can include a pocket, recess, or via to enable use of the multi-mode input device <NUM> as a pointer. In certain embodiments, the multi-mode input device <NUM> is comprised of soft materials for ease of use and gripping ergonomics. In operation of the 3D mouse configuration, a hand of the user can be perpendicular to a tactile surface, such as a surface of a desk or table. The multi-mode input device <NUM> is configured to transition between modes with minimal to no change of a grip of the user. The multi-mode input device <NUM> provides a natural transition between modes via a simple rotation of the multi-mode input device <NUM> by the hand of the user.

The multi-mode input device <NUM> also includes a first set of buttons <NUM>. The first set of buttons <NUM> is positioned to be accessible by one or more fingers on a hand of the user while the user's hand is resting upon the multi-mode input device <NUM>, such as, when the user is operating the multi-mode input device <NUM> in a 2D mode, similar to operating a computing mouse, as shown in the example illustrated in <FIG>. The multi-mode input device <NUM> also includes a second set of buttons <NUM>. In certain embodiments, the multi-mode input device <NUM> includes a touch pad <NUM>. The touch pad <NUM> can be a capacitive trackpad. The second set of buttons <NUM> and/or the touch pad <NUM> are positioned to be accessible by a thumb on the hand of the user while the user is gripping the multi-mode input device <NUM> in their hand, such as, when the user is operating the multi-mode input device <NUM> in a 3D mode, such as a VR/AR/MR/XR controller, as shown in the example illustrated in <FIG>.

In certain embodiments, one or more of the first set of buttons <NUM> are configured to be engaged while the multi-mode input device <NUM> is in the 3D mode illustrated in <FIG>. In certain embodiments, one of more of the second set of buttons <NUM> and/ or the touch pad <NUM> are configured to be engaged while the multi-mode input device <NUM> is in the 2D mode illustrated in <FIG>.

The multi-mode input device <NUM> includes processing circuitry, such as processor <NUM>, a memory <NUM>, an input/output (IO) interface <NUM>, a communication interface <NUM>, or sensors <NUM>. The multi-mode input device <NUM> can include a bus <NUM> configured to couple one or more of the processor <NUM>, memory <NUM>, IO interface <NUM>, communication interface <NUM>, or sensors <NUM>.

The processor <NUM> includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> is able to perform control on at least one of the other components of the multi-mode input device <NUM>, and/or perform an operation or data processing relating to an operational mode of the multi-mode input device <NUM>.

The memory <NUM> can include a volatile and/or non-volatile memory. For example, the memory <NUM> can store commands or data related to operation and orientation of the multi-mode input device <NUM>.

In certain embodiments, the multi-mode input device <NUM> includes one or more sensors <NUM>. The sensors <NUM> include one or more sensors configured to measure, detect, or determine a position or an orientation of the multi-mode input device <NUM>. For example, the multi-mode input device <NUM> can include one or more sensors <NUM> such as one or more of: one or more optic sensors, infrared sensors, an accelerometer, gyroscopic sensor, a magnetometer, a grip-position sensor, or an orientation sensor system. The orientation sensor system can be a virtual sensor that uses one or more algorithms to combine information from one or more of: an accelerometer, gyroscopic sensor, or a magnetometer. In certain embodiments, the magnetometer is configured to perform 6DOF electromagnetic field tracking. In certain embodiments, the multi-mode input device <NUM> includes an electromagnetic coil to track position and control modes. In certain embodiments, the multi-mode input device <NUM> includes an inertial measurement unit (IMU) to track position and control modes. In certain embodiments, the multi-mode input device <NUM> includes a machine vision circuit, such as one or more visible or infrared (IR) emitters and/or receivers, to track position and control modes. In certain embodiments, the multi-mode input device <NUM> includes a plurality of touch sensors and/or switches to determine hand/grip position of a user to control modes. In certain embodiments, the multi-mode input device <NUM> uses a combination of sensors, IMU, electromagnetic coils, and machine vision circuit to track position and control modes. In certain embodiments, the multi-mode input device <NUM> includes a combined positioned sensor configured to detect motions in the 2D and 3D plane. For example, the combined positioned sensor can be configured to detect that the multi-mode input device <NUM> resting on a tactile surface and detect lateral movements of the multi-mode input device <NUM> when the multi-mode input device <NUM> is operating as a computer mouse. Additionally, the combined position sensor can be configured to detect that the multi-mode input device <NUM> has been lifted from the tactile surface and thereafter detect 6DOF motions in space, such as when the multi-mode input device <NUM> is operating as a controller. In certain embodiments, the 3D position sensor comprises a plurality of video cameras configured to measure a position of the input device relative to an initial position. That is, the system is configured to detect and identify 3D motion based on a compared succession of images or a compared current position to a previous or initial position.

The IO interface <NUM> can include one or more of a data input buttons or a touch pad. In certain embodiments, the touch pad includes a capacitive trackpad. In certain embodiments, the IO interface <NUM> includes a mode switch. The mode switch is configured to be engaged by the user to switch the multi-mode input device <NUM> from a first mode including a 2D mode, such as a computing mouse, as shown in the example illustrated in <FIG>, and a second mode including a 3D mode, such as a VR/ AR/MR/XR controller ("ontroller"), as shown in the example illustrated in <FIG>.

The computer interface <NUM> includes a communication circuit, such as a transmitter or transceiver, configured to communicate orientation data and command data to an electronic device to which the multi-mode input device <NUM> is coupled. In certain embodiments, the multi-mode input device <NUM> communicates orientation data to the electronic device <NUM>. For example, the multi-mode input device <NUM> can communicate indicator information, such as a 2D indicator indicating that the multi-mode input device <NUM> is operating in a 2D mode or a 3D indicator indicating that the multi-mode input device <NUM> is operating in a 3D mode.

The processor <NUM> is configured to determine an orientation of the multi-mode input device <NUM>. In certain embodiments, the processor <NUM> receives positional data, orientation data, or both from one or more sensors <NUM>. In certain embodiments, the processor <NUM> detects a switch position or receives a signal indicating a switch position. Therefore, processor <NUM> is configured to perform position and operation mode detection. Through the one or more sensors <NUM>, the switch (IO interface <NUM>), or both, the processor <NUM> is configured to detect a position of the multi-mode input device <NUM>, a gesture by the multi-mode input device <NUM>, a hand activity corresponding to the multi-mode input device <NUM>, a physical input of the multi-mode input device <NUM>. The position can include one or more of: elevation, orientation, or specific areas of movement. The gesture can include ore or more of: acceleration or movement in one or more patterns by the multi-mode input device <NUM>. The hand activity can include a first activity in which the multi-mode input device <NUM> is held in a fashion similar to that in which a computer mouse is held, a second activity in which the multi-mode input device <NUM> is held in a fashion similar to which a 3D controller is held, or an absent position in which no hand is present. The physical input can include engagement of a specified button, a switch, a tap region, a grip-position sensor, or the like. In certain embodiments, the processor <NUM> is configured detect the orientation as a function of the application being executed on the electronic device <NUM>. That is, in certain embodiments, the 2D mode / 3D mode switching is application specific. In certain embodiments, the processor <NUM> is configured to perform one or more of: hand presence detection, hand position sensing for virtual representation of human interaction, or hand position sensing for grip-dependent functionality and mode switching.

Therefore, the multi-mode input device <NUM> is configured to provide: a contoured housing having ergonomics affording seamless 2D and 3D grip; hand presence detection; hand position sensing for virtual representation of human interaction; hand position sensing for grip-dependent functionality and mode switching. The multi-mode input device <NUM> is configured to enable 3D tracking for virtual representations of the physical device and 2D and 3D gesture support. In certain embodiments, based on communications between the electronic device <NUM> and the multi-mode input device <NUM>, the electronic device <NUM> is configured to provide virtual representations, such as mouse position and orientation, cursor or icon contextual shapes, and hand position representations. The mouse position and orientation enables or assists a user to find the multi-mode input device <NUM> and enhances intuition and use. The hand representations include indications regarding which buttons respective fingers of the user are on or near or representations of a hand indicating how the device is held within the hand. The multi-mode input device <NUM> includes a mouse and icon (pointer or shape) that provide for personal customization, contextual shapes (such as being more arrow shaped when in a "pointer" mode) and color highlights indicating actional areas, such as active buttons or an active trackpad.

<FIG> illustrate another multi-mode input device according to embodiments of the present disclosure. The embodiment of the multi-mode input device <NUM> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. The multi-mode input device <NUM> can be the same as, or include similar features as, the multi-mode input device <NUM>. For example, the multi-mode input device <NUM> can include one or more of: a processor <NUM>, memory <NUM>, IO interface <NUM>, communication interface <NUM>, sensors <NUM>, or bus <NUM>.

The multi-mode input device <NUM> includes a contoured housing <NUM>. The multi-mode input device <NUM> also includes one or more of: one or more input buttons <NUM>, a touch pad <NUM>, toggle, or thumb button, or a scroll wheel <NUM>.

The contoured housing <NUM> includes a curved top surface <NUM>. The curved top surface <NUM> is dimensioned to fit comfortably when a hand of the user is resting on the curved top surface <NUM>. That is, the curved top surface <NUM> is ergonomically designed to fit within a palm of the hand of the user when the hand of the user is resting upon the multi-mode input device <NUM>. The contoured housing <NUM> also includes a bottom surface <NUM> that is substantially flat. That is, the bottom surface is flat but may include one or more glides <NUM> that may protrude from the flat bottom surface <NUM>. Additionally, the contoured housing <NUM> may include one or more indentions, such as configured to enable access to a battery compartment or a charge port. The bottom surface <NUM> may also include an optic sensor <NUM>, or track ball, configured to detect or measure lateral motion of the multi-mode input device <NUM> along a tactile surface, such as a surface of a desk or table. Therefore, the bottom surface <NUM> is flat, but may have one or more protrusions or indentions, thus rendering the bottom surface <NUM> substantially flat. The contoured housing <NUM> is dimensioned to have a width <NUM> configured to enable the multi-mode input device <NUM> to comfortable fit between a thumb and a fifth digit, also known as a "little finger" or "pinky" (namely, the most ulnar and smallest finger of a hand, opposite the thumb, and next to the ring finger), of a hand of the user. The contoured housing <NUM> also includes a recess <NUM> on a lateral edge of the contoured housing <NUM>. The recess <NUM> can extend from the bottom surface <NUM> to at least half way up the lateral surface toward the curved top surface <NUM>. The recess <NUM> is dimensioned to be ergonomically comfortable to a user when a hand of the user is gripping the multi-mode input device <NUM> and provides ergonomics for 2D use and 3D gripping. For example, the recess <NUM> can be dimensioned to accommodate one or more of the fourth digit (namely, the "ring finger") and the fifth digit of the hand of the user while the user is holding the multi-mode input device <NUM> within the hand of the user. That is, when the user grips the multi-mode input device <NUM> and lifts the multi-mode input device <NUM> away from a tactile surface, such as a surface of a desk or table, a hand of the user envelopes and grips the multi-mode input device <NUM> such that one or more fingers of the hand engage, contact, or fit partially within the recess <NUM>.

<FIG> illustrates another multi-mode input device according to embodiments of the present disclosure. The embodiment of the multi-mode input device <NUM> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. The multi-mode input device <NUM> can be the same as, or include similar features as, the multi-mode input device <NUM> or multi-mode input device <NUM>. For example, the multi-mode input device <NUM> can include one or more of: a processor <NUM>, memory <NUM>, IO interface <NUM>, communication interface <NUM>, sensors <NUM>, or bus <NUM>.

The multi-mode input device <NUM> includes a contoured housing <NUM>. The multi-mode input device <NUM> also includes one or more of: one or more input buttons <NUM>, a touch pad <NUM>, toggle, or thumb button, or a scroll wheel (not shown).

The contoured housing <NUM> includes a top surface. The top surface is dimensioned to fit comfortably when a hand of the user is resting on the top surface. That is, the top surface is ergonomically designed to fit within a palm of the hand of the user when the hand of the user is resting upon the multi-mode input device <NUM>. The contoured housing <NUM> also includes a bottom surface that is substantially flat. That is, the bottom surface is flat but may include one or more glides that may protrude from the flat bottom surface. Additionally, the contoured housing <NUM> may include one or more indentions, such as configured to enable access to a battery compartment or a charge port. The bottom surface may also include an optic sensor, or track ball, configured to detect or measure lateral motion of the multi-mode input device <NUM> along a tactile surface, such as a surface of a desk or table. Therefore, the bottom surface is flat, but may have one or more protrusions or indentions, thus rendering the bottom surface substantially flat. The contoured housing <NUM> is dimensioned such that a width <NUM> of the multi-mode input device <NUM> enables the multi-mode input device <NUM> to comfortably fit between a thumb and a fifth digit of a hand of the user. The contoured housing <NUM> also includes an arced opening or via <NUM>. A circumferential edge of the via <NUM> includes a portion formed by the contoured housing <NUM> and a portion formed by a switch <NUM>. The via <NUM> is dimensioned to enable the contoured housing <NUM> to be ergonomically comfortable to a user when a hand of the user is gripping the multi-mode input device <NUM>. For example, the via <NUM> can be dimensioned to accommodate one or more of the fourth digit (namely, the "ring finger") and the fifth digit of the hand of the user while the user is holding the multi-mode input device <NUM> within the hand of the user. That is, when the user grips the multi-mode input device <NUM> and lifts the multi-mode input device <NUM> away from a tactile surface, such as a surface of a desk or table, a hand of the user envelopes and grips the multi-mode input device <NUM> such that one or more fingers of the hand engage the switch <NUM> and fit partially within the via <NUM>. For example, as a user picks-up the multi-mode input device <NUM>, one or more fingers engage a lever arm of the switch <NUM>, depressing the lever arm inwards toward the via <NUM> enabling the user to grip the contoured housing <NUM> and lever arm of the switch <NUM> within the hand of the user.

The processor, such as processor <NUM>, is able to detect that the switch <NUM> is engaged and determines that the multi-mode input device <NUM> is to operate in a 3D mode. For example, when the user picks-up the multi-mode input device <NUM> and engages (i.e., presses) the lever arm of the switch <NUM>, the switch <NUM> can transmits a signal to the processor <NUM> indicating that the multi-mode input device <NUM> is gripped in a hand of the user and is to operate in the 3D mode as a 3D controller. In certain embodiments, the processor <NUM> detects that the lever arm of the switch <NUM> has been engaged. For example, the lever arm of the switch <NUM> may close an electrical circuit to which the processor <NUM> also is coupled. When the lever arm of the switch <NUM> is engaged, the movement of the lever arm opens the circuit. Thereafter, the processor <NUM> detects the open circuit, such as by detecting an interruption in a current flow.

<FIG> illustrates another multi-mode input device according to embodiments of the present disclosure. The embodiment of the multi-mode input device <NUM> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. The multi-mode input device <NUM> can be the same as, or include similar features as, the multi-mode input device <NUM>, the multi-mode input device <NUM> or the multi-mode input device <NUM>. For example, the multi-mode input device <NUM> can include one or more of: a processor <NUM>, memory <NUM>, IO interface <NUM>, communication interface <NUM>, sensors <NUM>, or bus <NUM>.

The contoured housing <NUM> includes a top surface. The top surface is dimensioned to fit comfortably when a hand of the user is resting on the top surface. That is, the top surface is ergonomically designed to fit within a palm of the hand of the user when the hand of the user is resting upon the multi-mode input device <NUM>. The contoured housing <NUM> also includes a bottom surface that is substantially flat. That is, the bottom surface is flat but may include one or more glides that may protrude from the flat bottom surface. Additionally, the contoured housing <NUM> may include one or more indentions, such as configured to enable access to a battery compartment or a charge port. The bottom surface may also include an optic sensor, or track ball, configured to detect or measure lateral motion of the multi-mode input device <NUM> along a tactile surface, such as a surface of a desk or table. Therefore, the bottom surface is flat, but may have one or more protrusions or indentions, thus rendering the bottom surface substantially flat. The contoured housing <NUM> is dimensioned such that a width <NUM> of the multi-mode input device <NUM> enables the multi-mode input device <NUM> to comfortably fit between a thumb and a fifth digit of a hand of the user. The contoured housing <NUM> also includes a via <NUM>. The via <NUM> is dimensioned to enable the contoured housing <NUM> to be ergonomically comfortable to a user when a hand of the user is gripping the multi-mode input device <NUM>. For example, the via <NUM> can be dimensioned to accommodate one or more of the second digit (namely the "pointer finger"), the third digit (namely the "middle finger"), the fourth digit, and the fifth digit of the hand of the user while the user is holding the multi-mode input device <NUM> within the hand of the user. That is, when the user grips the multi-mode input device <NUM> and lifts the multi-mode input device <NUM> away from a tactile surface, such as a surface of a desk or table, a hand of the user envelopes and grips the multi-mode input device <NUM> such that one or more fingers fit partially within or through the via <NUM>.

<FIG> and <FIG> illustrates different multi-mode input devices according to embodiments of the present disclosure. The embodiments of the multi-mode input devices shown in <FIG> and <FIG> are for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Each of the multi-mode input device can be the same as, or include similar features as, the multi-mode input device <NUM>, the multi-mode input device <NUM>, the multi-mode input device <NUM>, or multi-mode input device <NUM>. For example, each of the multi-mode input devices shown in <FIG> and <FIG> can include a contoured housing and one or more of: a processor <NUM>, memory <NUM>, IO interface <NUM>, communication interface <NUM>, sensors <NUM>, bus <NUM>, touch pad, data buttons, recesses, or vias.

<FIG> illustrates a transition in operation mode from 2D to 3D according to embodiments of the present disclosure. The embodiment of the transition shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the examples disclosed in <FIG> reference multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In the example shown in <FIG>, a user is operating a multi-mode input device <NUM> in a 2D plane. That is, the multi-mode input device <NUM> is operated on a flat, tactile surface, such as a surface of a desk or table. The multi-mode input device <NUM> provides a 2D input to a connected computing device, such as electronic device <NUM>.

In the example shown in <FIG>, a user lifts the multi-mode input device <NUM> from the 2D plane into 3D space. For example, the user lifts the multi-mode input device <NUM> into the air space above the flat, tactile surface.

In the example shown in <FIG>, the user operates the multi-mode input device <NUM> in 3D space. That is, the multi-mode input device <NUM> can be moved about in the air. By moving the multi-mode input device <NUM> through space, the multi-mode input device <NUM> provides 3D input to the computing device.

<FIG> illustrate a transition of a cursor or object display from 2D to 3D according to embodiments of the present disclosure. The embodiment of the transition shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the examples disclosed in <FIG> reference multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In the example shown in <FIG>, the multi-mode input device <NUM> is operating in a 2D plane, such as shown in <FIG>. The multi-mode input device <NUM> provides, to the attached computing device, such as electronic device <NUM>, a planar position of multi-mode input device <NUM>. The computing device displays, such as on a display <NUM>, a 2D pointer <NUM> as a cursor. The 2D pointer <NUM> illustrates that the multi-mode input device <NUM> is operating in a 2D plane. The 2D pointer <NUM> also can provide a spatial position of the multi-mode input device <NUM>.

In the example shown in <FIG>, the multi-mode input device <NUM> is operating in a 3D space, such as shown in <FIG>. The multi-mode input device <NUM> provides, to the attached computing device, such as electronic device <NUM>, a spatial position of multi-mode input device <NUM>. In certain embodiments, the spatial position is based on 3DOF information. In certain embodiments, the spatial position is based on 6DOF information. The computing device displays, such as on a display <NUM>, a 3D object <NUM>. The 3D object <NUM> illustrates that the multi-mode input device <NUM> is operating in a 3D space. The 3D object <NUM> also can provide a spatial position and orientation of the multi-mode input device <NUM> via movement and rotation of the 3D object <NUM> when the multi-mode input device <NUM> is operating in 3D space.

<FIG> illustrates a process for using sensor data to automatically transition between modes according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks one or more sensors of the multi-mode input device <NUM>. The processor <NUM> reads, without an input or command received from the user, one or more measurements from sensor <NUM>. For example, the processor <NUM> may receive a measurement signal from one or more sensors indicating that the multi-mode input device <NUM> is flat or raised. If the processor <NUM> determines that the multi-mode input device <NUM> is flat, such as from the multi-mode input device <NUM> resting on the tactile surface, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines that the multi-mode input device <NUM> is raised, such the multi-mode input device <NUM> is lifted away from the tactile surface, the processor <NUM> enables a 3D operating mode in block <NUM>. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

<FIG> illustrates a process for using an electromagnetic coil sensor data to automatically transition between modes according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks one or more sensors of the multi-mode input device <NUM>. The processor <NUM> reads, without an input or command received from the user, one or more measurements from an electromagnetic coil system or electromagnetic coil sensor, such as sensor <NUM>. For example, the processor <NUM> may receive a measurement signal from the electromagnetic coil system indicating that the multi-mode input device <NUM> is flat or raised. If the processor <NUM> determines that the multi-mode input device <NUM> is flat, such as from the multi-mode input device <NUM> resting on the tactile surface, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines that the multi-mode input device <NUM> is raised, such the multi-mode input device <NUM> is lifted away from the tactile surface, the processor <NUM> enables a 3D operating mode in block <NUM>. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

<FIG> illustrates a process for using an inertial measurement unit (IMU) data to automatically transition between modes and track position according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks one or sensors <NUM>, such as an IMU, of the multi-mode input device <NUM>. The processor <NUM> reads, without an input or command received from the user, one or more measurements from the IMU. For example, the processor <NUM> may receive a measurement signal from the IMU indicating that the multi-mode input device <NUM> is flat or raised. If the processor <NUM> determines that the multi-mode input device <NUM> is flat, such as from the multi-mode input device <NUM> resting on the tactile surface, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines that the multi-mode input device <NUM> is raised, such the multi-mode input device <NUM> is lifted away from the tactile surface, the processor <NUM> enables a 3D operating mode in block <NUM>. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

<FIG> illustrates a process for using machine/computer vision data to automatically transition between modes and track position according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks one or sensors <NUM>, such as machine/ computer vision sensors, of the multi-mode input device <NUM>. The machine/computer vision may include one or more of visible light sensors, IR sensors, visible light detectors, or IR detectors. The processor <NUM> reads, without an input or command received from the user, one or more measurements from the machine/computer vision sensors. For example, the processor <NUM> may receive a measurement signal from the machine/computer vision sensors indicating that the multi-mode input device <NUM> is flat or raised. If the processor <NUM> determines that the multi-mode input device <NUM> is flat, such as from the multi-mode input device <NUM> resting on the tactile surface, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines that the multi-mode input device <NUM> is raised, such the multi-mode input device <NUM> is lifted away from the tactile surface, the processor <NUM> enables a 3D operating mode in block <NUM>. For example, the processor <NUM> can determine, based on measurements from the machine/computer vision sensors, that the multi-mode input device <NUM> has been raised to a certain elevation or distance from the tactile surface. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

<FIG> illustrates a process for using data from a combination of sensors to automatically transition between modes and track position according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks a combination of sensors <NUM>, such as machine/computer vision sensors, accelerometers, IMU, gyroscopes, magnetometers, of the multi-mode input device <NUM>. The processor <NUM> reads, without an input or command received from the user, one or more measurements from the combination of sensors <NUM>. For example, the processor <NUM> may receive a measurement signal from the combination of sensors <NUM> indicating that the multi-mode input device <NUM> is flat or raised. If the processor <NUM> determines that the multi-mode input device <NUM> is flat, such as from the multi-mode input device <NUM> resting on the tactile surface, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines that the multi-mode input device <NUM> is raised, such the multi-mode input device <NUM> is lifted away from the tactile surface, the processor <NUM> enables a 3D operating mode in block <NUM>. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

<FIG> illustrates a process for user hand position detection to automatically transition between modes according to embodiments of the present disclosure. The embodiment of the transition <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. Although the example disclosed in <FIG> references multi-mode input device <NUM>, embodiments with any of the multi-mode input devices <NUM>, <NUM>, or <NUM>, apply equally.

In block <NUM>, the processor <NUM> checks sensors <NUM>, such a physical contact sensor or switch, of the multi-mode input device <NUM>. The processor <NUM> reads, without an input or command received from the user, one or more measurements from the sensors <NUM>. For example, the processor <NUM> may receive a measurement signal from the sensors <NUM> indicating that a hand of the user is one of: absent, holding the multi-mode input device <NUM> in a mouse-grip (i.e., hand resting upon a top surface of the multi-mode input device <NUM>, or gripping the multi-mode input device <NUM> in a pointer grip (i.e., hand gripping the multi-mode input device <NUM> such that one or more fingers are enveloped around a portion of the multi-mode input device <NUM>). If the processor <NUM> determines that the hand is absent from the multi-mode input device <NUM>, the processor <NUM> disables the multi-mode input device <NUM> and enables an idle mode in block <NUM>. If the processor <NUM> determines that the hand is holding the multi-mode input device <NUM> in a mouse-grip, such as when the multi-mode input device <NUM> resting on the tactile surface and the hand is atop the multi-mode input device <NUM>, the processor <NUM> enables a 2D operating mode in block <NUM>. In the 2D operating mode, the processor <NUM> enables a mouse input and disables a controller input. If the processor <NUM> determines the hand is gripping the multi-mode input device <NUM> in a pointer grip, such as when the multi-mode input device <NUM> is lifted away from the tactile surface and held within a hand of the user, the processor <NUM> enables a 3D operating mode in block <NUM>. In the 3D operating mode, the processor <NUM> enables a 3D controller input and disables a 2D mouse input.

Accordingly, embodiments of the present disclosure provide for several mode switching mechanisms. The multi-mode input device can switch based on one or more of: position, gestures, hand activity, software context, or physical input. Position switching includes detection and action in response to one or more of: elevation, orientation, and movement into specific areas. Gesture switching includes acceleration patterns, motion patterns, and the like. Hand activity switching includes pointer gripping, mouse-type holding, or absence. Software context includes application specific switching, on-button responses, and the like. Physical input switching includes button pressing and depressing, switch engagement, tapping, and the like.

Embodiments of the present disclosure provide a system having an electronic device and a multi-mode input device <NUM>. The system is configured to provide virtual representations, such as mouse position and orientation, cursor or icon contextual shapes, and hand position representations. The mouse position and orientation enables or assists a user to find the multi-mode input device <NUM> and enhances intuition and use. The hand representations include indications regarding which buttons respective fingers of the user are on or near or representations of a hand indicating how the device is held within the hand. The multi-mode input device <NUM> includes a mouse and icon that provide for personal customization, contextual shapes (such as being more arrow shaped when in a "pointer" mode) and color highlights indicating actionable areas, such as active buttons or an active trackpad.

Although various features have been disclosed in particular embodiments, aspects of each embodiment are not exclusive to the disclosed embodiment and may be combined without other embodiments without departing from the scope of the present disclosure. That is, one or more features disclosed in first embodiment may be combined with one or features from a second or third embodiment without departing from the scope of the present disclosure.

Although various features have been shown in the figures and described above, various changes may be made to the figures. For example, the size, shape, arrangement, and layout of components shown in <FIG> are for illustration only. Each component could have any suitable size, shape, and dimensions, and multiple components could have any suitable arrangement and layout. Further, each component in a device or system could be implemented using any suitable structure(s) for performing the described function(s). In addition, while <FIG> illustrates various series of steps, various steps in <FIG> could overlap, occur in parallel, occur multiple times, or occur in a different order.

Claim 1:
An input device (<NUM>; <NUM>; <NUM>; <NUM>) comprising:
a position sensor (<NUM>); and
a processor (<NUM>) configured to:
switch between operation modes as a function of orientation measurements detected by the position sensor;
enter a three-dimensional (3D) input mode when the processor (<NUM>) detects that the input device (<NUM>; <NUM>; <NUM>; <NUM>) is in a first orientation, and
enter a two-dimensional (2D) input mode when the processor (<NUM>) detects that the input device (<NUM>; <NUM>; <NUM>; <NUM>) is in a second orientation,
wherein the 2D input mode is configured to provide 2D position data to a connected computer system, and wherein the 3D input mode is configured to provide 3D position data to the connected computer system;
characterised in that the input device (<NUM>; <NUM>; <NUM>; <NUM>) further comprises:
a first set of buttons (<NUM>) positioned to be accessible by one or more fingers on a hand of a user while the hand of the user is resting upon the input device, when the user is operating the input device in the 2D input mode;
a second set of buttons (<NUM>) positioned to be accessible by a thumb on the hand of the user while the hand of the user is gripping the input device, when the user is operating the input device in the 3D input mode; and
a touch pad (<NUM>) positioned to be accessible by a thumb on the hand of the user while the hand of the user is gripping the input device, when the user is operating the input device in the 3D input mode.