Patent ID: 12242768

DETAILED DESCRIPTION

The present disclosure provides systems and methods for collaborating physical-virtual interfaces. Human inhabited characters may include, for example, avatars, animations, virtual characters, or holograms, among other types of images/embodiments used for real time (or live) interaction between a human that inhabits the character and another human inhabited character. In such live interactions, the human inhabited character may be controlled or embodied by a human user, sometimes called a human specialist (interchangeably referred to herein as a specialist, simulation specialist, inhabitor, or interactor). A human inhabited character is suited for emotionally complex and sensitive human interactions that cannot be replicated in a scripted artificial intelligence (AI)-driven system. Inclusion of a human in a virtual environment to control the character in real time allows another human, such as a trainee (interchangeably referred to herein as learner, end user, or person to be counseled), to experience a wider range of human emotions and non-verbal cues in an individualized experience not limited by scripted responses. The human inhabited character may respond like a real human being in real time.

Typically, a specialist may request one or more trainees to collaborate on work (e.g., solve problem, answer question) by displaying the work within the virtual environment. While conferencing systems and other software tools provide the ability to share whiteboards, these types of systems and tools may not allow shared collaborative work while trainees are immersed in virtual environment while wearing, for example, a headset (or head mounted display). In the absence of an interface between the physical and the virtual collaborative workspaces, attempts to share and exchange data may detract from an immersive experience (e.g. one can remove the headset to write on a physical surface and put the headset back on again).

According to aspects of the present disclosure, a collaborative workspace may be shared between interactive devices and within a virtual environment such that users may interact with each other in real-time. Aspects of the present disclosure also allow human-inhabited characters to respond to input from an interactive device by changing an action or appearance of the human inhabited character thereby providing an immersive environment for users. Aspects of the present disclosure further provide techniques for users of virtual reality (VR) devices, such as head mounted displays (HMDs), to interact with interactive devices while simultaneously using the VR devices.

Turning now to the figures, examples of systems and methods for collaborating physical-virtual interfaces on a computing system are depicted. It is to be understood that aspects of the figures may not be drawn to scale and are instead drawn for illustrative purposes.

Referring toFIG.1, an example system100includes a collaboration controller102communicatively coupled one or more interactive systems104to via a network106. Example implementations of the system may include, but are not limited to, shared collaborative problem solving (e.g., educational teaching or professional training), medical simulations (e.g., collaboration on patient charts or electrocardiogram (EKG or ECG)), or conferencing systems (e.g., shared tools for collaboration such as white-board markups).

In an example, the collaboration controller102may be a computing device that communicates with a specialist (not shown) to host and provide an interactive virtual environment190. In an example, the collaboration controller102may synchronize data between a plurality of the interactive systems104such that collaborative workspaces are shared. Examples of the collaboration controller102may include a server, a computer, a laptop, a tablet, or any computing device configured to organize and manage the interactive virtual environment190. The collaboration controller102may include artificial intelligence (AI) or a neural network trained to correlate inputs from one or more interactive devices110with action states or appearance states of one or more human-inhabited characters192of a virtual environment190. Examples of the network106includes one or more of a local area network, wide area network, or the Internet, among other types of networks.

Each of the interactive systems104may include an interactive device110communicatively coupled with a corresponding immersive device120. In an aspect, the interactive device110and the virtual device120may communicate via wireless communications (e.g., Bluetooth® or WiFi) or wired communications.

The interactive device110may include a display112for displaying digital content180. In an example, the digital content180may include collaborative work provided by the specialist. The digital content180may include documents, images, or digital media stored on the interactive device110and/or received from the collaboration controller102. In an aspect, the interactive device110may be configured to enable a user to interact with the digital content180. For example, the interactive device110may include one or more digital editing tools that receives input from a user170of the interactive device110to markup, manipulate, and/or revise the digital content180. The digital editing tools may include pens, pencils, highlighters, erasers, etc. As a non-limiting example, the interactive device110may receive content markup182to answer a question (e.g., mathematical question) on the digital content180. Examples of an interactive device110may include a tablet, a laptop, a smartphone, or any other computing device for interacting with digital content180.

As described herein, the collaboration controller102may synchronize data between a plurality of the interactive systems104but in an alternative example, two or more interactive systems104may synchronize data between each other without the use of the collaboration controller102.

The immersive device120may include a display122for displaying the virtual environment190including one or more human-inhabited characters192or virtual objects such as a virtual device194or a virtual white-board196. In an example, the virtual device194displayed by the display122may represent a corresponding interactive device110. Examples of the immersive device120include head mounted displays (HMDs), laptops, tablets, or any other computing device configured to display a virtual environment. The immersive device120may include one or more input devices, such as a camera or a microphone, to allow a user to communicate with other users. Alternatively, or in addition, one or more interactive devices110may include one or more of these input devices for communication between users.

The interactive system104may also be configured to control verbal and non-verbal characteristics of a human-inhabited character192. For example, input received by the interactive device110may be interpreted by the collaboration controller102to illustrate a non-verbal characteristic including an action state such as moving a hand or an arm of the human-inhabited character192in a writing motion on the virtual device194or the virtual white-board196to show the human inhabited character192is entering the content markup182. Further, the collaboration controller102may transition other non-verbal characteristics including an appearance state (e.g., posture, facial expressions) of the human-inhabited character192due to the input from the interactive device110. In an aspect, the collaboration controller102may use this input in combination with other inputs including, for example, visual or verbal behaviors captured by a camera or microphone of the interactive system104.

Some interactive systems104may include a controller124for controlling the corresponding interactive device110and/or the corresponding immersive device120(e.g., HMD). In an example, the controller124may include one or more input components such as buttons, paddle sticks, or touchpads for receiving input from a user and location circuitry including, but not limited to accelerometers or gyroscopes, for determining a position, orientation, or rotation of the controller124. In an aspect, the controller124may be physically connected to the corresponding interactive device110such that the position, orientation, or rotation of the controller124may be inherited by the virtual device194in the virtual environment190. The physical connection between the controller124and the interactive device110may allow the virtual device194to be seen and interacted with in the virtual environment190by the user170without having to remove the immersive device120(e.g., HMD). Further, input received via the input components of the controller124may be received by the interactive device110and used to markup or manipulate the digital content180displayed on the interactive device110.

As discussed herein, the digital content180and the content markup182may be synchronized such that the content markup182from a first interactive device110is displayed on one or more of second interactive devices110and/or in the virtual environment190of one or more of the immersive device120(e.g., in virtual device194and/or virtual white-board196).

Synchronizing Data

Referring toFIG.2, each markup action (e.g., content markup182) from any of the interactive devices110may be synchronized with other devices while still allowing each of the interactive devices110to continue to receive user input to simultaneously markup the digital content180. In an example, if the digital content180is stored on each interactive device110, a integer identifier may be sent from the collaboration controller102via the network106to the respective interactive device110to indicate which content to display. In the event the digital content180is not stored on one or more interactive devices110, the collaboration controller102may send the digital content180(e.g., a lossless image) to the interactive devices110. In an aspect, the digital content180may include location values indicating a location points of the digital content180based on a coordinate system (e.g., Cartesian coordinates).

To synchronize the content markups182, the interactive devices110may transmit each markup stroke to the collaboration controller102, where a markup stroke may consist of a starting point, a stopping point, and a time value, and, in some examples a color, or a tool (highlighter, pen, etc.) In an aspect, the starting point and the stopping point may be identified based on the location points of the digital content180. For example, a starting point may be identified by integers indicating x and y locations corresponding to the digital content180.

In an example, the markup strokes may be sent reliably, but without any guarantee of order. As a result, the time value (e.g., T1-T4) may be used to reconstruct the content markup182as the markup strokes from any of the interactive devices110may be received by the collaboration controller102out of order. In an example, the collaboration controller102stores a markup stroke and the last received time value for each pixel. When the collaboration controller102receives a new markup stroke at a more recent time, the collaboration controller102draws the markup stroke on a new texture. A shader (e.g., shader520ofFIG.5) can then be used to combine the new texture with the rest of the markup based on the digital editing tool that was used by the respective interactive device110. For example, a pen tool may simply be drawn on top of pixels with older time values and completely discarded on pixels with more recent time values. In another example, the highlighter tool may use both the time value and an alpha value of the original pixel to construct a markup.

As an example, as illustrated byFIG.2, the collaboration controller102may receive and store start and stop points for markup stroke202at time T1, markup stroke204at time T2, markup stroke206at time T3, and markup stroke208at time T4. As each of these markup strokes are received, the shader may combine these markup strokes to eventually form the content markup182.

Methods of Implementation

Referring toFIG.3, an example method300for collaborating physical-virtual interfaces is depicted. In an example, the method300may be performed by one or more components of the computing device500ofFIG.5, which is an example of one or more of the interactive device110or the immersive device120. Examples of some of the operations of the method300may be described in relation toFIGS.1and2.

At302, the method300may include transmitting, to one or more immersive devices, digital content to display in a virtual environment that includes one or more human inhabited characters, the digital content corresponding to content displayed on one or more interactive devices. For example, the collaboration controller102(or a processor502or a communications component506of the collaboration controller102) may transmit, to the immersive devices120, the digital content180to display in the virtual environment190that includes the human inhabited characters192, the digital content180corresponding to content displayed on the interactive devices110.

At304, the method300may include receiving, from a first interactive device of the one or more interactive devices, first input data representing content markup of the digital content from a first user of the first interactive device. For example, the collaboration controller102(or a processor502of the collaboration controller102) may receive, from the interactive device110, the content markup182(or any one of markup strokes202-208) representing content markup of the digital content180from the user170of the interactive device110.

At306, the method300may include determining an action state or an appearance state for a human inhabited character of the one or more human inhabited characters in response to the first data. For example, the collaboration controller102(or a processor502of the collaboration controller102) may determine an action state or an appearance state for the human inhabited character192in response to the content markup182. In an example, the action state may include an action (such as writing) of the human inhabited character corresponding to the content markup182, and an appearance state may include a facial expression (such as looking at a virtual device that is being written on) of the human inhabited character192corresponding to the content markup182.

At308, the method300may include transmitting, to the one or more immersive devices, the human inhabited character to display in the virtual environment according to the action state or the appearance state. For example, the collaboration controller102(or a processor502or a communications component506of the collaboration controller102) may transmit, to the immersive devices120, the human inhabited character192to display in the virtual environment190according to the action state or the appearance state.

Referring toFIG.4, another example method400for collaborating physical-virtual interfaces is depicted. In an example, the method400may be performed by one or more components of the computing device500ofFIG.5, which is an example of one or more of the interactive device110or the immersive device120. Examples of some of the operations of the method400may be described in relation toFIGS.1and2.

At402, the method400may include receiving, from a first interactive device of one or more interactive devices, first input data representing content markup of digital content from a first user of the first interactive device, the digital content to be display in a virtual environment that includes one or more human inhabited characters. For example, the collaboration controller102(or the processor502or the communications component506of the collaboration controller102) may receive the markup stroke202(e.g., first input data) from the interactive device110, wherein the markup stroke202represents content markup of the digital content180. In one or more other examples, the first input data may include any of the markup strokes202-208, and or a combination of one or more of these markup strokes.

At404, the method400may include determining one or more points corresponding to the content markup and a time value of the content markup. For example, the collaboration controller102(or processor502of the collaboration controller102) may determine one or more of a starting point or a stopping point of the markup stroke202and a time value of the markup stroke202. In an example, the starting point or the stopping point may be integer values corresponding to a location of the points on the digital content180.

At406, the method400may include synchronizing display of the first input data across the one or more interactive devices and the virtual environment of one or more immersive devices based on the one or more points and the time value. For example, the collaboration controller102(or processor502of the collaboration controller102) may synchronize display of the markup stroke202across the one or more interactive devices110and the virtual environment190of one or more immersive devices120based on the one or more points and the time value.

Referring toFIG.5, illustrated is an example of components of a computing device500in accordance with an implementation. The computing device500may be an example of one or more of the collaboration controller102, the interactive device110, or the immersive device120. In an example, the computing device500may include the processor502for carrying out processing functions associated with one or more of components and functions described herein. In an example, the processor502may also include a neural network to perform one or more operations described herein. The processor502may include a single or multiple set of processors or multi-core processors. Moreover, the processor502may be implemented as an integrated processing system and/or a distributed processing system.

In an example, the computing device500may include a memory504for storing instructions executable by the processor502for carrying out the functions described herein.

Further, the computing device500may include the communications component506that provides for establishing and maintaining communications with one or more parties utilizing hardware, software, and services as described herein. The communications component506may carry communications between components on the computing device500, as well as between the computing devices500and/or external devices and devices located across a communications network and/or devices serially or locally connected to the computing device500. For example, the communications component506may include one or more buses, and may further include transmit chain components and receive chain components associated with a transmitter and receiver, respectively, operable for interfacing with external devices.

The computing device500may also include a graphics processor508operable to render content for display on the computing device500. The graphics processor508may include a shader520for rendering images on a display.

The computing device500may also include a user interface component510operable to receive inputs from a user of the computing device500and further operable to generate outputs for presentation to the user. The user interface component510may include one or more input devices including but not limited to a keyboard, a number pad, a camera, a mouse, an input controller, a touch-sensitive display530(e.g., display112or122), a digitizer, a navigation key, a function key, a microphone, a voice recognition component, any other mechanism capable of receiving an input from a user, or any combination thereof. Further, the user interface component510may include one or more output devices, including but not limited to a display530(e.g., display112or122), a speaker, a haptic feedback mechanism, a printer, any other mechanism capable of presenting an output to a user, or any combination thereof.

FIG.6is a block diagram of various example system components, in accordance with an aspect of the present disclosure.FIG.6illustrates a communication system600usable in accordance with aspects of the present disclosure. The communication system600may include one or more accessors660,662(e.g., user170) and one or more terminals642,666(e.g., interactive device110, immersive device120, computing device500). In an aspect, data for use in accordance with aspects of the present disclosure is, for example, input and/or accessed by accessors660,662via terminals642,666, such as personal computers (PCs), minicomputers, mainframe computers, microcomputers, telephonic devices, or wireless devices, such as personal digital assistants (“PDAs”) or a hand-held wireless devices coupled to a server643(e.g., collaboration controller102), such as a PC, minicomputer, mainframe computer, microcomputer, or other device having a processor and a repository for data and/or connection to a repository for data, via, for example, a network644(e.g., network106), such as the Internet or an intranet, and couplings645,646,664. The couplings645,646,664include, for example, wired, wireless, or fiberoptic links. In another example variation, the method and system in accordance with aspects of the present disclosure operate in a stand-alone environment, such as on a single terminal.

As used in this application, the terms “component,” “system” and the like are intended to include a computer-related entity, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer device and the computer device can be a component. One or more components can reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets, such as data from one component interacting with another component in a local system, distributed system, and/or across a network such as the Internet with other systems by way of the signal.

Moreover, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.

Various implementations or features may have been presented in terms of systems that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. discussed in connection with the figures. A combination of these approaches may also be used.

The various illustrative logics, logical blocks, and actions of methods described in connection with the embodiments disclosed herein may be implemented or performed with a specially-programmed one of a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computer devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Additionally, at least one processor may comprise one or more components operable to perform one or more of the steps and/or actions described above.

Further, the steps and/or actions of a method or procedure described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor, such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. Further, in some implementations, the processor and the storage medium may reside in an ASIC. Additionally, the ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal. Additionally, in some implementations, the steps and/or actions of a method or procedure may reside as one or any combination or set of codes and/or instructions on a machine readable medium and/or computer readable medium, which may be incorporated into a computer program product.

In one or more implementations, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

While implementations of the present disclosure have been described in connection with examples thereof, it will be understood by those skilled in the art that variations and modifications of the implementations described above may be made without departing from the scope hereof. Other implementations will be apparent to those skilled in the art from a consideration of the specification or from a practice in accordance with examples disclosed herein.