Switching between views using natural gestures

The disclosure includes a system and method for switching between video views and data views. The system includes a controller, a view presentation module, a screen detection module and a view switching module. The controller receives data indicating a participant joined a multi-user communication session. The view presentation module presents a video stream on a mobile device associated with the participant. The screen detection module determines an occurrence of a detection trigger event. The controller receives a video frame image responsive to the occurrence of the detection trigger event. The screen detection module detects a data screen in the video frame image. The view switching module switches a view on the mobile device from video view to data view responsive to a natural gesture performed by the participant. The view presentation module presents a data stream associated with the data screen on the mobile device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The specification relates to a system and method for switching between video views and data views. In particular, the specification relates to a system and method for switching between video views and data views using natural gestures in mobile videoconferencing.

2. Description of the Background Art

Existing videoconferencing systems collect and transmit data streams along with video and audio streams in a videoconference session. This is because in most business meetings, the users expect to not only see each other, but also to exchange data information, such as documents, presentation slides, handwritten comments, etc. These data streams are usually directly captured from computer screens, separately encoded with special coding tools, and displayed side-by-side with the video streams on a remote site.

The explosion of mobile devices drives more and more videoconferencing service providers to develop mobile applications such as smart phone and tablet applications. These mobile applications make it much easier for the users to access the videoconferencing service from anywhere using mobile devices.

However, it becomes a problem to display both a video view and a data view on the mobile device simultaneously. Due to the limited screen size of the mobile device, it is not possible to display both the video view and the data view at full resolution side-by-side. Currently, the commonly used method is to use a user interface that displays one view at the full screen scale while showing only the thumbnail for the other view. The user interface combines and displays the video view and the data view together. Such a user interface fails to provide a unified experience by separating the user interface into multiple view modes, and may cause confusion when there is more than one data stream.

SUMMARY OF THE INVENTION

The disclosure includes a system and method for switching between video views and data views on a mobile device. In one embodiment, the system includes a controller, a view presentation module, a screen detection module and a view switching module. The controller receives data indicating a participant joins a multi-user communication session. The view presentation module presents a video stream of the multi-user communication session on a mobile device associated with the participant. The screen detection module determines an occurrence of a detection trigger event. The controller receives a video frame image from the video stream responsive to the occurrence of the detection trigger event. The screen detection module detects a first data screen in the video frame image. The controller receives data describing a first natural gesture performed on the mobile device. The view switching module switches a view on the mobile device from video view to data view responsive to the first natural gesture. The view presentation module presents a first data stream associated with the first data screen on the mobile device.

In another embodiment, a computer-implemented method with the following steps is performed. The method receives data indicating that a first participant, a second participant and a third participant joined a multi-user communication session. The method presents a video stream of a multi-user communication session to a mobile device associated with the third participant. The method receives a video frame image from the first video stream that includes a first device associated with the first participant and a second device associated with the second participant. The method detects a first data screen from the first device and a second data screen from the second device in the video frame image. The method receives data describing a selection of the first data screen performed on the mobile device. The method switches a view on the mobile device from video view to a first data view that corresponds to the first data screen responsive to the selection. The method presents the first data stream on the mobile device.

Other aspects include corresponding methods, systems, apparatuses, and computer program products for these and other innovative aspects.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The system described in the disclosure is particularly advantageous in numerous respects. First, the system allows a participant to use natural gestures such as pinch gestures to switch between video views and data views, and is capable of providing consistent and seamless user experience in multi-user communication sessions including mobile videoconferencing sessions.

Second, the system is capable of automatically detecting data screens in video frame images. A participant sees a data screen in a video view before switching to a data view showing the data screen in full resolution, which allows the participant to understand the relationship between a data stream of the data screen and the video content in the video view and therefore avoids confusion when more than one data stream is present. For example, if a speaker in a conference room moves frequently between two or more data screens such as a projector screen and a whiteboard screen, the system can eliminate a remote viewer's confusion between the projector screen and the whiteboard screen when the remote viewer frequently switches between the video view and the projection screen data view or between the video view and the whiteboard screen data view.

Third, the system supports embedded data streams and is capable of providing embedded data streams to users. For example, the system can present a data stream of a current meeting in a data view mode, where the data stream of the current meeting is a video clip describing a meeting. The video clip is embedded with slides and whiteboard stoke information presented in the previous meeting. The system can switch from the data view mode to the embedded data view mode to present the embedded slides and whiteboard stroke information to participants of the current meeting in full resolution. The system may have other numerous advantages.

A system and method for switching between video views and data views is described below. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the embodiments can be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the invention. For example, the invention is described in one embodiment below with reference to mobile devices such as a smart phone and particular software and hardware. However, the description applies to any type of computing device that can receive data and commands, and any peripheral devices providing services.

System Overview

FIG. 1Aillustrates a block diagram of a system100for switching between video views and data views according to one embodiment. The illustrated system100includes a hosting device101accessible by a host135, a registration server130, a camera103, display devices107a. . .107nand mobile devices115a. . .115naccessible by participants125a. . .125n. InFIG. 1Aand the remaining figures, a letter after a reference number, e.g., “115a,” represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., “115,” represents a general reference to instances of the element bearing that reference number. In the illustrated embodiment, these entities of the system100are communicatively coupled via a network105.

The network105can be a conventional type, wired or wireless, and may have numerous different configurations including a star configuration, token ring configuration or other configurations. Furthermore, the network105may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and/or other interconnected data paths across which multiple devices may communicate. In some embodiments, the network105may be a peer-to-peer network. The network105may also be coupled to or includes portions of a telecommunications network for sending data in a variety of different communication protocols. In some embodiments, the network105includes Bluetooth communication networks or a cellular communications network for sending and receiving data including via short messaging service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, WAP, email, etc. AlthoughFIG. 1Aillustrates one network105coupled to the mobile devices115, the hosting device101and the registration server130, in practice one or more networks105can be connected to these entities.

A hosting environment137can be an environment to host a multi-user communication session. An example multi-user communication session includes a videoconferencing meeting. In some examples, a hosting environment137is a room where all the devices within the dashed box inFIG. 1Aare visible to users. For example, the hosting environment137could be a conference room environment including one or more display devices107and one or more cameras103present in the conference room. Example display devices107include, but are not limited to, a projector, an electronic whiteboard, a liquid-crystal display and any other conventional display devices. In one embodiment, the camera103is an advanced videoconferencing camera. Example cameras103include, but are not limited to, a high-definition (HD) video camera that captures high-resolution videos, a pan-tilt-zoom (PTZ) camera that can be mechanically controlled or a group of cameras that provide multi-view or panoramic views in the hosting environment137. Although two display devices107and one camera103are illustrated inFIG. 1A, the hosting environment137can include one or more display devices107and one or more cameras103.

The hosting device101, the display devices107a. . .107nand the camera103are located within the hosting environment137. The hosting device101is communicatively coupled to the display device107avia signal line116, the display device107nvia signal line118and the camera103via signal line114. The display device107ais optionally coupled to the registration server130via signal line102; the display device107nis optionally coupled to the registration server130via signal line104; and the camera103is optionally coupled to the registration server130via signal line112.

The hosting device101can be a computing device that includes a processor and a memory, and is coupled to the network105via signal line131. For example, the hosting device101is a hardware server. In another example, the hosting device101is a laptop computer or a desktop computer. The hosting device101is accessed by a host135, for example, a user that manages a meeting. The hosting device101includes a hosting application109and a storage device for storing the presentations generated by the hosting application109.

The hosting application109includes software for hosting a multi-user communication session. For example, the hosting application109hosts a video conferencing meeting that the host135manages and one or more participants125join using one or more mobile devices115. In another example, the hosting application109generates slides for giving a presentation.

In one embodiment, the hosting application109displays data to be shared with other participants125on one or more data screens of one or more display devices107in the hosting environment137. Data to be shared with participants125includes, but is not limited to, a text-based document, web page content, presentation slides, video clips, stroke-based handwritten comments and/or other user annotations, etc. The one or more data screens in the hosting environment137are visible to the camera103. For example, the presentation slides to be shared with other remote participants125are projected on the wall, where the projection of the presentation slides (or, at least a predetermined portion of the projection) is within a field of view of the camera103. In this case, the camera103is capable of capturing the projection of the presentation slides in one or more video frame images of a video stream.

In another example, if a user in a conference room writes comments on an electronic whiteboard, the hosting application109can control movement of the camera103so that the electronic whiteboard is visible to the camera103. In this case, the camera103is capable of capturing the comments shown in the electronic whiteboard in one or more video frame images of a video stream. The camera103captures a video stream including video frame images depicting the hosting environment137, where the video frame images contain data screens of the display devices107and/or the data screen of the hosting device101.

In one embodiment, the camera103sends the video stream to the hosting device101, causing the hosting device101to forward the video stream to one or more of the registration server130and the mobile device115. In another embodiment, the camera103sends the video stream directly to the registration server130and/or the mobile device115via the network105. In yet another embodiment, the camera103sends a latest video frame image captured by the camera103to the registration server130responsive to an occurrence of a detection trigger event. The detection trigger event is described below in more detail with reference toFIG. 2.

In one embodiment, the hosting application109or the display device107captures a high quality version of a data stream displayed on a data screen of the display device107. This high quality version of the data stream displayed on the data screen is referred to as a data stream associated with the data screen, which includes a series of data screen images (e.g., screenshot images) depicting content displayed on the data screen of the display device107over time. A screenshot image of a data screen depicts content displayed on the data screen at a particular moment of time. At different moments of time, different screenshot images of the data screen are captured, which form a data stream associated with the data screen. In some examples, a screenshot image of the data screen may be also referred to as a data frame of the data stream.

In some examples, the hosting application109captures a series of screenshot images describing a slide presentation in high resolution directly from a presentation computing device. In some additional examples, an electronic whiteboard captures original stroke information displayed on the whiteboard screen, and sends screenshot images depicting the original stroke information to the hosting application109.

In one embodiment, the hosting application109sends the data stream associated with the data screen to one or more of the mobile device115and the registration server130. In another embodiment, the display device107directly sends the data stream including one or more data screen images to one or more of the mobile device115and the registration server130. For example, the display device107periodically sends an up-to-date data screen image to the registration server130.

In one embodiment, the participation application123acan be operable on the registration server130. The registration server130includes a processor and a memory, and is coupled to the network105via signal line106. The registration server130includes a database for storing registered images. The registration server130registers the display device107and receives a video feed for a meeting from the camera103. The video feed includes one or more video frame images. The registration server130runs image matching algorithms to find a correspondence between a latest video frame and a latest screenshot image of a data screen associated with the display device107or the hosting device101. If a match is found, the matching area is highlighted in the video frame image and displayed on the mobile device115. The registration server130is described below in more detail with reference toFIGS. 2 and 3A.

In another embodiment, the participation application123bmay be stored on a mobile device115a, which is connected to the network105via signal line108. The mobile device115a,115nis a computing device with limited display space that includes a memory and a processor, for example a laptop computer, a tablet computer, a mobile telephone, a smartphone, a personal digital assistant (PDA), a mobile email device or other electronic device capable of accessing a network105. The mobile device115includes a touch screen for displaying data and receiving natural gestures from a participant125. Examples of natural gestures include, but are not limited to, tap, double tap, long press, scroll, pan, flick, two finger tap, pinch open, pinch close, etc.

In the illustrated embodiment, the participant125ainteracts with the mobile device115a. The mobile device115nis communicatively coupled to the network105via signal line110. The participant125ninteracts with the mobile device115n. The participant125can be a remote user participating in a multi-user communication session such as a videoconferencing session hosted by the hosting device101. The mobile devices115a,115ninFIG. 1Aare used by way of example. WhileFIG. 1Aillustrates two mobile devices115aand115n, the disclosure applies to a system architecture having one or more mobile devices115.

In one embodiment, the participation application123is distributed such that it may be stored in part on the mobile device115a,115nand in part on the registration server130. For example, the participation application123bon the mobile device115acts as a thin-client application that displays the video stream or the data stream while the registration server130performs the screen detection steps. The participation application123bon the mobile device115ainstructs the display to present the video stream or the data stream, for example, by rendering images in a browser. The participation application123breceives user input (e.g., natural gestures) from the participant125and interprets the user input. For example, assume the participation application123bcurrently displays the video stream. The participation application123breceives user input from the participant125ato magnify the screen so much that it overcomes a threshold and the participation application123bdetermines that the stream should be switched to the data stream of the hosting device101. The participation application123bsends instructions indicating to switch from the video stream to the data stream to the participation application123aon the registration server130.

The participation application123can be code and routines for participating in a multi-user communication session. In one embodiment, the participation application123can be implemented using hardware including a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In another embodiment, the participation application123can be implemented using a combination of hardware and software. In various embodiments, the participation application123may be stored in a combination of the devices and servers, or in one of the devices or servers.

FIG. 1Bis another embodiment of a system for switching between video views and data views. In this embodiment, there is no hosting device101. Instead, mobile devices115can comprise the camera103and the participation application123b. The mobile devices115are coupled to the display devices107a,107nvia signal lines136,138, respectively.

The participant125can activate the camera103on the mobile device115and point it at the display devices107a,107nto capture their content. The mobile device115can transmit the images directly to the registration server130via signal line154. For example, the images can serve as a query from the mobile device115. The participation application123uses the captured images to detect the screen from the video view and switches to the data view in response to receiving gestures from the participant125.

Example Participation Application

Referring now toFIG. 2, an example of the participation application123is shown in more detail.FIG. 2is a block diagram of a computing device200that includes a participation application123, a processor235, a memory237, an input/output device241, a communication unit239and a storage device243according to some examples. The components of the computing device200are communicatively coupled by a bus220. The input/output device241is communicatively coupled to the bus200via signal line242. In some embodiments, the computing device200can be one of a mobile device115and a registration server130. For example, in one embodiment, the registration server130can include a participation application123with some of the components described below and the mobile device115can include some of the other components described below.

The processor235includes an arithmetic logic unit, a microprocessor, a general purpose controller or some other processor array to perform computations and provide electronic display signals to a display device. The processor235is coupled to the bus220for communication with the other components via signal line222. Processor235processes data signals and may include various computing architectures including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture implementing a combination of instruction sets. AlthoughFIG. 2includes a single processor235, multiple processors235may be included. Other processors, operating systems, sensors, displays and physical configurations are possible.

The memory237stores instructions and/or data that can be executed by the processor235. The memory237is coupled to the bus220for communication with the other components via signal line224. The instructions and/or data may include code for performing the techniques described herein. The memory237may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory or some other memory device. In some embodiments, the memory237also includes a non-volatile memory or similar permanent storage device and media including a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a more permanent basis.

The communication unit239transmits and receives data to and from at least one of the hosting device101, the mobile device115and the registration server130depending upon where the participation application123is stored. The communication unit239is coupled to the bus220via signal line226. In some embodiments, the communication unit239includes a port for direct physical connection to the network105or to another communication channel. For example, the communication unit239includes a USB, SD, CAT-5 or similar port for wired communication with the mobile device115. In some embodiments, the communication unit239includes a wireless transceiver for exchanging data with the mobile device115or other communication channels using one or more wireless communication methods, including IEEE 802.11, IEEE 802.16, BLUETOOTH® or another suitable wireless communication method.

In some embodiments, the communication unit239includes a cellular communications transceiver for sending and receiving data over a cellular communications network including via short messaging service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, WAP, e-mail or another suitable type of electronic communication. In some embodiments, the communication unit239includes a wired port and a wireless transceiver. The communication unit239also provides other conventional connections to the network105for distribution of files and/or media objects using standard network protocols including TCP/IP, HTTP, HTTPS and SMTP, etc.

The storage device243can be a non-transitory memory that stores data for providing the functionality described herein. The storage device243may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory or some other memory devices. In some embodiments, the storage device243also includes a non-volatile memory or similar permanent storage device and media including a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a more permanent basis.

In the illustrated embodiment, the storage device243is communicatively coupled to the bus220via signal line228. In one embodiment, the storage device243stores one or more of a video stream including one or more video frame images, a data stream including one or more data screen images and one or more detection trigger events, etc. The storage device243may store other data for providing the functionality described herein. For example, the storage device243could store copies of video conferencing materials, such as presentations, documents, audio clips, video clips, etc.

In the illustrated embodiment shown inFIG. 2, the participation application123includes a controller202, a view presentation module204, a screen detection module206, a view switching module208, a user interface module210and an optional camera adjustment module212. The components of the participation application123are communicatively coupled via the bus220. Persons of ordinary skill in the art will recognize that the components can be stored in part on the mobile device115and in part on the registration server130. For example, the participation application123stored on the registration server130could include the screen detection module206and the participation application123stored on the mobile device could include the remaining components.

The controller202can be software including routines for handling communications between the participation application123and other components of the computing device200. In one embodiment, the controller202can be a set of instructions executable by the processor235to provide the functionality described below for handling communications between the participation application123and other components of the computing device200. In another embodiment, the controller202can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the controller202can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line230.

In one embodiment, the controller202sends and receives data, via the communication unit239, to and from one or more of the mobile device115, the hosting device101and the registration server130. For example, the controller202receives, via the communication unit239, user input from a participant125operating on a mobile device115and sends the user input to the view switching module208. In another example, the controller202receives graphical data for providing a user interface to a participant125from the user interface module210and sends the graphical data to a mobile device115, causing the mobile device115to present the user interface to the participant125.

In one embodiment, the controller202receives data from other components of the participation application123and stores the data in the storage device243. For example, the controller202receives data describing one or more detection trigger events from the screen detection module206and stores the data in the storage device243. In another embodiment, the controller202retrieves data from the storage device243and sends the data to other components of the participation application123. For example, the controller202retrieves a data stream from the storage device243and sends the data stream to the view presentation module204for presenting the data stream to a participant125.

The view presentation module204can be software including routines for presenting a video view or a data view on a mobile device115. In one embodiment, the view presentation module204can be a set of instructions executable by the processor235to provide the functionality described below for presenting a data view or a video view on a mobile device115. In another embodiment, the view presentation module204can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the view presentation module204can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line232.

A video view mode presents video data associated with a multi-user communication session to a participant125. For example, the video view mode presents the video stream of the other participants in the multi-user communication session to the participant125in full screen on the mobile device115. In another example, the video view mode presents the video stream on the mobile device115in full resolution.

In one embodiment, the view presentation module204receives data indicating that a participant125joins a multi-user communication session from a mobile device115associated with the participant125. Correspondingly, the mobile device115is in the video view mode. The view presentation module204receives a video stream including one or more video frame images from the camera103directly or via the hosting device101, and presents the video stream to the participant125on a display of the mobile device115.

In some examples, one or more data screens that are in the same hosting environment137as the camera103are captured in the one or more video frame images of the video stream, and the one or more video frame images include sub-images depicting the one or more data screens. For example, the one or more video frame images capture at least a portion of a data screen of a hosting device101, a portion of a screen projection on the wall and/or a portion of a data screen of an electronic whiteboard. In another example, the one or more video frame images capture the full data screen of the hosting device101, the full projection screen on the wall and/or the full data screen of the electronic whiteboard.

A data view mode presents a data stream associated with the multi-user communication session to the participant125. For example, the data view mode presents a data stream with the slides being presented during the multi-user communication session to the participant125in full screen on the mobile device115. In another example, the data view mode presents the data stream on the mobile device115in full resolution.

In one embodiment, the view presentation module204receives, from the view switching module208, an identifier (ID) of a detected data screen and a view switching signal indicating that a view on the mobile device115should be switched from the video view to the data view. In some embodiments, the view presentation module204receives a data stream associated with the detected data screen directly from the display device107associated with the data screen. In some other embodiments, the view presentation module204receives the data stream via the hosting device101. Responsive to receiving the view switching signal, the view presentation module204stops presenting the video stream on the mobile device115and starts to present the data stream associated with the data screen on the mobile device115. Persons of ordinary skill in the art will recognize that the sections describing presenting the video stream or data stream are meant to represent the view presentation module204instructing the user interface module210to generate graphical data that is sent to the mobile device115via the communication unit239for display.

In some examples, an embedded data stream is included in the data stream. The view presentation module204receives, from the view switching module208, a view switching signal instructing the view presentation module204to switch the view on the mobile device115from the data view to an embedded data view. The embedded data view mode presents the embedded data stream to the participant125in full resolution or in full screen on the mobile device115. Responsive to receiving the view switching signal, the view presentation module204stops presenting the data stream and starts to present the embedded data stream on the mobile device115. An embedded data stream can be a videoconferencing meeting, a presentation, a video clip, a text document, presentation slides, or other types of data embedded in the data stream.

If the view presentation module204receives a view switching signal instructing the view presentation module204to switch the view from the embedded data view back to the data view from the view switching module208, the view presentation module204stops presenting the embedded data stream and starts to present the data stream on the mobile device115again. In one embodiment, the view presentation module204receives a view switching signal instructing the view presentation module204to switch the view on the mobile device115from the data view to the video view from the view switching module208. Responsive to the view switching signal, the view presentation module204stops presenting the data stream and starts to present the video stream on the mobile device115.

The screen detection module206can be software including routines for performing data screen detection in a video frame image. In one embodiment, the screen detection module206can be a set of instructions executable by the processor235to provide the functionality described below for performing data screen detection in a video frame image. In another embodiment, the screen detection module206can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the screen detection module206can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line234.

In one embodiment, the screen detection module206registers one or more display devices107with the registration server130. For example, the screen detection module206can record a device identifier, a user associated with the display device107, etc. with the display device107and store the registration information in the storage243. Each display device107sends an updated image of its data screen to the registration server130periodically. For example, each display device107sends its up-to-date screenshot image to the registration server130periodically. In some examples, the display device107sends the updated screenshot images of its data screen to the registration server130via the hosting device101.

In one embodiment, the screen detection module206detects an occurrence of a trigger event. For example, the event could be a detection trigger event that triggers a detection of one or more data screens in a video frame image. For example, a detection trigger event causes the screen detection module206to detect whether the video frame image includes a data screen. Example detection trigger events include, but are not limited to, motion of the camera103(e.g., panning, zooming or tilting of the camera103, movement of the camera103, etc.) and/or motion of an object in the video frame image (e.g., appearance of a projection on the wall in the video frame image, movement of a whiteboard, etc.). In another example, the trigger event could be based on a timer.

Responsive to the occurrence of the detection trigger event, the screen detection module206receives a latest video frame image of the video stream from the camera103directly or via the hosting device101. In some examples, the screen detection module206receives the latest video frame image of the video stream from the mobile device115or a video server that provides the video stream. The screen detection module206performs data screen detection in the latest video frame image responsive to the occurrence of the detection trigger event. For example, the screen detection module206determines whether a data screen appears in the latest video frame image by matching a latest screenshot image of the data screen with the latest video frame image.

In some examples, for each data screen registered with the registration server130, the screen detection module206determines whether a sub-image that matches the latest screenshot image of the data screen appears in the latest video frame image. For example, the screen detection module206determines whether the latest video frame image includes a sub-image that depicts the data screen (e.g., the screen detection module206determines whether the data screen is captured by the latest video frame image). In a further example, the screen detection module206runs an image matching algorithm to find the correspondence between the latest video frame image and the latest screenshot image of the data screen. If the screen detection module206finds a match between the latest video frame image and the latest screenshot image of the data screen, the screen detection module206highlights the matching area in the video frame image on the mobile device115. For example, the screen detection module206highlights the detected data screen in the video frame image on the mobile device115.

In one embodiment, the screen detection module206runs the image matching algorithm in real time. An example image matching algorithm includes a scale-invariant feature transform (SIFT) algorithm. The SIFT algorithm extracts feature points of both the latest video frame image and the latest screenshot image of the data screen, where the feature points from both images are matched based on the k-nearest neighbors (KNN), and the random sample consensus (RANSAC) algorithm is used to find the consensus and to determine the homographic matrix. Additional information about how to use SIFT, KNN and RANSAC for image matching can be found at Hess, R.,An Open-Source SIFT Library, Proceedings of the International Conference on Multimedia, October 2010, pp. 1493-96. Persons of ordinary skill in the art will recognize that other image matching algorithms can be used.

If the screen detection module206detects one or more data screens existing in the video frame image, the screen detection module206generates a matching result including one or more matches between the one or more data screens and the video frame image. The screen detection module206notifies the mobile device115of the one or more matches, and establishes a direct connection between the mobile device115and each display device107that has one matched data screen. The screen detection module206highlights one or more matching areas in the video frame image, where each matching area corresponds to a position of one data screen captured in the video frame image. The screen detection module206displays the highlighted matching areas on the mobile device115.

In another embodiment, the camera103is statically deployed and captures one or more data screens in the hosting environment137, and positions of the one or more data screens remain unchanged in the video frame images. The screen detection module206can determine existence of the one or more data screens based on the static setting in the hosting environment137, and can pre-calibrate positions of the one or more data screens in the video frame images. The screen detection module206highlights the one or more data screens in the video frame images at the pre-calibrated positions in the video frame images.

The screen detection module206sends one or more screen IDs identifying the one or more detected data screens and data describing one or more matching areas in the video frame image to the view switching module208. In another embodiment, the screen detection module206sends pre-calibrated positions of one or more data screens to the view switching module208. In yet another embodiment, the screen detection module206stores the one or more screen IDs, data describing the one or more matching areas and/or the pre-calibrated positions in the storage243.

The view switching module208can be software including routines for switching a view on a mobile device115between a video view and a data view. In one embodiment, the view switching module208can be a set of instructions executable by the processor235to provide the functionality described below for switching a view on a mobile device115between a video view and a data view. In another embodiment, the view switching module208can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the view switching module208can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line236.

In one embodiment, the view switching module208receives, from the screen detection module206, data describing one or more screen IDs identifying one or more detected data screens and one or more matching areas associated with the one or more detected data screens in the video frame image. In the video view mode, the mobile device115presents the video stream to the participant125, with the one or more detected data screens highlighted in the matching areas of the video frame images. If the participant125performs a natural gesture (e.g., a pinch open or double tap gesture, etc.) within a highlighted matching area of a data screen on a touch screen of the mobile device115, the view switching module208interprets the participant's natural gesture as a command to switch from the video view to the data view. The view switching module208generates a view switching signal describing the command and sends the view switching signal to the view presentation module204, causing the view presentation module204to present the data view to the participant125. In one embodiment, the view switching module208interprets the natural gesture as a command to switch from the video view to the data view if the portion of the data screen detected in the video frame image is greater than a predetermined threshold (e.g., a majority portion of the data screen appearing in the video frame image).

For example, the participant125can use a natural gesture to zoom into a data screen detected in the video frame image, so that the video view presenting the video frame image scales up accordingly on the touch screen of the mobile device115. If the size of the scaled-up data screen in the video frame image reaches a predetermined threshold, the view switching module208automatically switches the view on the mobile device115from the video view to the data view, causing the view presentation module204to present the data stream associated with the detected data screen on the mobile device115. The mobile device115switches from the video view mode to the data view mode accordingly. The participant125can further perform natural gestures to operate on the data stream such as zooming into the data stream, copying the data stream, dragging the data stream, etc.

In the data view mode, the mobile device115presents the data stream to the participant125. If the participant125performs a natural gesture (e.g., a pinch close gesture or tapping on an exit icon, etc.) on the data stream displayed on a touch screen of the mobile device115, the view switching module208interprets the participant's natural gesture as a command to switch from the data view back to the video view. The view switching module208generates a view switching signal describing the command and sends the view switching signal to the view presentation module204, causing the view presentation module204to present the video view to the participant125. Again, the screen detection module206detects the one or more data screens visible to the camera103in the video frame images, and highlights the one or more data screens in the video frame images. For example, the participant125can use a natural gesture to zoom out the data stream, so that the data view presenting the data stream scales down accordingly on the touch screen of the mobile device115. If the size of the scaled down data stream reaches a predetermined threshold, the view switching module208automatically switches the view on the mobile device115from the data view to the video view, causing the view presentation module204to present the video stream on the mobile device115.

In the data view mode, if the presented data stream includes an embedded data stream, the participant125can use a natural gesture on the embedded data stream. The view switching module208interprets the natural gesture as a command to switch from the data view to the embedded data view. The view switching module208generates a view switching signal describing the command and sends the view switching signal to the view presentation module204, causing the view presentation module204to present the embedded data stream to the participant125in full resolution. The participant125may perform another natural gesture to exit from the embedded data view and return to the data view. For example, if the data stream includes an embedded video, the participant125can issue a tap open command on an icon representing the embedded video during the data view mode, causing the view presentation module204to present the embedded video in full screen on the mobile device115. After viewing the embedded video, the participant125can issue a pinch close command to exit from the embedded data view and return to the data view.

The user interface module210can be software including routines for generating graphical data for providing a user interface. In one embodiment, the user interface module210can be a set of instructions executable by the processor235to provide the functionality described below for generating graphical data for providing a user interface. In another embodiment, the user interface module210can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the user interface module210can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line238.

In one embodiment, the user interface module210receives instructions from the view presentation module204to generate graphical data for providing a user interface to a user such as a host135or a participant125. The user interface module210sends the graphical data to the hosting device101or the mobile device115, causing the hosting device101or the mobile device115to present the user interface to the user. For example, the user interface module210generates graphical data for providing a user interface that depicts a video stream or a data stream. The user interface module210sends the graphical data to the mobile device115, causing the mobile device115to present the video stream or the data stream to the participant125via the user interface. In other embodiments, the user interface module210may generate graphical data for providing other user interfaces to users.

The optional camera adjustment module212can be software including routines for adjusting a camera103. In one embodiment, the camera adjustment module212can be a set of instructions executable by the processor235to provide the functionality described below for adjusting a camera103. In another embodiment, the camera adjustment module212can be stored in the memory237of the computing device200and can be accessible and executable by the processor235. In either embodiment, the camera adjustment module212can be adapted for cooperation and communication with the processor235and other components of the computing device200via signal line240.

In one embodiment, a participant125can use natural gestures to navigate the camera103. For example, the participant125can perform a natural gesture to change the view angle of the camera103via a user interface shown on the mobile device115. The camera adjustment module212receives data describing the participant's natural gesture and interprets the participant's natural gesture as a command to adjust the camera103such as panning, tilting, zooming in or zooming out the camera103. The camera adjustment module212adjusts the camera103according to the participant's natural gesture. Through adjusting the camera103, the participant125may keep one or more data screens of one or more display devices107within the field of view of the camera103, so that the camera103captures the one or more data screens in the video frame images.

An example use of the system described herein includes a videoconferencing scenario, where a first party (e.g., a host135) is in a conference room equipped with a camera103and one or more data screens, and a second party (e.g., a participant125) is a remote mobile user participating the videoconference using a mobile device115such as a smart phone or a tablet. After the participant125joins the videoconference, the participation application123receives a video stream from the camera103and presents the video stream to the participant125on a touch screen of the mobile device115. The participation application123detects one or more data screens captured by video frame images. The participant125can issue a natural gesture such as a pinch open gesture on a detected data screen highlighted in the video frame images, causing the mobile device115to switch from video view to data view. Afterwards, the participation application123presents a data stream associated with the detected data screen to the participant125in full resolution. The participant125may issue another natural gesture such as a pinch close gesture to switch from the data view back to the video view.

Another example use of the system described herein includes a retrieval application for retrieving information relevant to an image. For example, a user can capture an image of an advertisement (e.g., an advertisement for a vehicle brand), and instruct the retrieval application to retrieve information relevant to the advertisement. The image of the advertisement may include a banner and/or a data screen image showing a commercial video. The retrieval application can instruct the screen detection module206to detect the data screen in the image of the advertisement and to identify a product that matches content shown in the data screen image. The retrieval application may retrieve information relevant to the identified product from one or more databases and provide the relevant information to a user. Other example uses of the system described herein are possible.

Graphic Representations

FIG. 3Ais a graphic representation300illustrating one embodiment of a process for performing data screen detection. After a participant125joins a multi-user communication session using the mobile device115, the camera103establishes a video stream connection302with the mobile device115. The camera103sends a video stream to the mobile device115via the video stream connection302, causing the mobile device115to present the video stream to the participant125in a video view mode. The display device107registers with the registration server130and sends updated screenshot images304of a data screen associated with the display device107to the registration server130periodically. In the illustrated example, the display device107is an electronic whiteboard. In one embodiment, the registration server130detects a detection trigger event. For example, the registration server130detects motion of the camera103such as panning or tilting. The registration server130receives a latest video frame image306from the camera103responsive to the detection trigger event. In some examples, the registration server130receives the latest video frame image306from the mobile device115.

The registration server130uses an image-matching method to detect active data screens dynamically. For example, the registration server130uses an image matching algorithm to find the correspondence between the latest video frame image306and the latest screenshot image received from either the hosting device101or the display device107. If a matching result308between the latest video frame image306and the latest screenshot image of the data screen is found, the registration server130notifies the mobile device115of the matching result308and highlights the corresponding data screen in the video frame images. For example, the registration server130uses a box310to highlight a data screen of an electronic whiteboard in the video frame image. The display device107associated with the data screen establishes a data stream connection312with the mobile device115. The display device107may send a data stream to the mobile device115via the data stream connection312.

FIG. 3Bis a graphic representation319illustrating one embodiment for switching between video views and data views on a mobile device115using natural gestures. The participation application123interprets natural gestures from a participant125to achieve the seamless user experience. In this embodiment, the participation application123captures and transmits a first data stream and a second data stream along with the video stream captured from the camera103to the mobile device115. The first data stream includes the high quality screenshot images from the hosting device101(e.g., a laptop), and the second data stream includes the strokes from the display device107(e.g., an electronic whiteboard). Both data screens (the data screen of the laptop and the data screen of the electronic whiteboard) are visible to the camera103. In one embodiment, the screenshot images from the hosting device101include an embedded image depicting the data screen of the display device107.

At the beginning, the video view is shown on the mobile device115to present the video frame image320to the participant125. For example, the video frame image320is shown in full resolution or on full screen of the mobile device115. As shown inFIG. 3B, both data screens (the data screen324of the laptop and the data screen322of the electronic whiteboard) are visible in the video frame image320displayed on the participant's mobile device115. The participation application123intelligently detects and notifies the participant125of the existence of data screens in the video frame images. For example, the participation application123highlights the data screens322and324in the video frame image320.

At phase (1), if the participant125tries to get more detail from the laptop data screen324, he or she can perform a natural gesture330on the laptop data screen324shown in the video frame image320to zoom into the laptop data screen324. An example natural gesture330can be a pinch or double tap gesture. Responsive to the natural gesture330, the video view on the mobile device115scales up. If the size of the recognized laptop data screen324reaches a pre-set threshold, the mobile device115automatically switches from the video view to the data view. For example, the view on the mobile device115switches from presenting the video frame image320in full resolution to presenting a high quality screenshot image326of the laptop data screen324in full resolution. The participation application123interprets any further pinch or dragging gestures performed on the screenshot image326as operating on the screenshot image326of the laptop data screen324.

At phase (2), when the participant125performs a natural gesture332such as a pinch gesture on the screenshot image326to zoom out of the data view and the zoom-out scale ratio reaches a pre-set threshold, the mobile device115switches back to the video view from the data view. Again, the participation application123presents the video frame image320on the mobile device115in full resolution, and detects and marks the visible data screens322and324in the video frame image320.

At phase (3), the participant125performs a natural gesture334such as a dragging gesture on the highlighted data screen322in the video frame image320, causing the mobile device115to enlarge the video view greater than a threshold amount, which causes the mobile device115to switch from showing the video frame image320in full resolution to the data view showing a screenshot image328of the electronic whiteboard in full resolution. At phase (4), the participant125performs a natural gesture336such as a pinch gesture on the screenshot image328to zoom out the data view, causing the mobile device115to decrease the data view until a threshold point triggers the mobile device115to switch back to the video view from the data view. Again, the mobile device115presents the video frame image320to the participant125.

FIG. 4Ais a graphic representation400of one embodiment of a graphic user interface illustrating a video view on a mobile device115. The example user interface shows a video frame image402depicting a conference room. The video frame image402depicts a host135and a data screen404of the hosting device101projected on a wall of the conference room. The data screen404includes an embedded data screen406. If the participant125performs a natural gesture on the data screen404captured in the video frame image402, the mobile device115switches from the video view to the data view shown inFIG. 4B.

FIG. 4Bis a graphic representation420of one embodiment of a graphic user interface illustrating a data view on a mobile device115. In this example, a data stream including screenshot images of the data screen404is presented on the mobile device115. The data stream is a multi-user communication session including an embedded data stream. For example, the data stream is a video clip of another conference with embedded slides. The embedded data screen406presenting the embedded slides is shown in the screenshot image of the data screen404.

When the participant125switches to the data view shown inFIG. 4Bfrom the video view shown inFIG. 4A, the data stream including the video clip starts to play. In one embodiment, the participant125may exit from the data view shown inFIG. 4Band return to the video view shown inFIG. 4Aby performing a natural gesture (e.g., a pinch to close gesture) on the screenshot image of the data screen404. In one embodiment, the participant125can keep zooming into the data view if the video clip includes embedded presentation slides or whiteboard strokes information. For example, if the participant125performs a natural gesture on the embedded data screen406, the mobile device115can switch from the data view to an embedded data view shown inFIG. 4Cto present slides embedded in the video clip.

FIG. 4Cis a graphic representation440of one embodiment of a graphic user interface illustrating an embedded data view on a mobile device115. In this example, the slides shown in the embedded data screen406is presented to the participant125. The participant125may exit from the embedded data view and return to the data view shown inFIG. 4Bby performing a natural gesture (e.g., a pinch to close gesture) on the screenshot image of the embedded data screen406.

Methods

FIG. 5is a flow diagram illustrating one embodiment of a method500for switching between video views and data views using natural gestures in a multi-user communication session. In one embodiment, the controller202receives502data indicating that a participant125joined a multi-user communication session from a mobile device115associated with the participant125. The view presentation module204presents504a video stream of the multi-user communication session to the mobile device115. For example, the view presentation module204instructs the user interface engine210to generate graphical data for displaying the video stream. In one embodiment, the screen detection module206determines an occurrence of a detection trigger event. The controller202receives506a video frame image from the video stream responsive to the occurrence of the detection trigger event. For example, the controller202receives a latest video frame image of the video stream from the camera103. The screen detection module206detects508a first data screen in the video frame image. For example, the screen detection module206determines that the video frame image captures the first data screen.

The controller202receives510data describing a first natural gesture performed on the mobile device115. For example, the controller202receives data describing a pinch to open gesture performed on the first data screen in the video frame image. The view switching module208switches512a view on the mobile device115from video view to data view responsive to the first natural gesture. The view presentation module204presents514a first data stream associated with the first data screen on the mobile device115. In one embodiment, the first data stream includes one or more high-definition screenshot images of the first data screen generated by a display device107associated with the first data screen.

FIGS. 6A-6Care flow diagrams illustrating another embodiment of a method600for switching between video views and data views using natural gestures in a multi-user communication session. Referring toFIG. 6A, the controller202receives602data indicating that a participant125joined a multi-user communication session from a mobile device115associated with the participant125. The view presentation module204presents604a video stream of the multi-user communication session on the mobile device115. The screen detection module206registers606a display device107with the registration server130. The display device107includes a data screen for presenting a data stream of the multi-user communication session in the hosting environment137. The controller202receives608images of the data screen from the display device107periodically. For example, the controller202receives screenshot images of the data screen from the display device107periodically.

The screen detection module206detects610an occurrence of a detection trigger event. The controller202receives612a latest video frame image from the camera103responsive to the occurrence of the detection trigger event. The screen detection module206performs614data screen detection in the latest video frame image using the latest image of the data screen received from the display device107.

Referring toFIG. 6B, the screen detection module206determines616whether a sub-image that matches the latest image of the data screen is found in the latest video frame image. If the sub-image is found in the latest video frame image, the method600moves to step618. Otherwise, the method600ends. Turning to step618, the screen detection module206generates a matching result indicating the match between the latest image of the data screen and the latest video frame image, and notifies the mobile device115of the matching result. The screen detection module206provides data between the mobile device115and the display device107associated with the data screen. For example, the screen detection module206establishes a direct connection between the devices. In one embodiment, the display device107can transmit a data stream associated with the data screen to the mobile device115via the direct connection.

The controller202receives622data describing a first natural gesture performed by the participant125on the sub-image depicting the data screen in the video frame image. The controller202receives624a data stream associated with the data screen from the display device107. The view switching module208switches626a view on the mobile device115from video view to data view responsive to the first natural gesture exceeding a threshold. For example, the user makes an expanding view starting in the center of the screen and moving over half the width of the screen. The view presentation module204presents628the data stream associated with the data screen on the mobile device115.

Referring toFIG. 6C, the controller202receives630data describing a second natural gesture performed by the participant125on the data stream. The view switching module208switches632the view on the mobile device115from data view back to video view responsive to the second natural gesture exceeding a threshold. The view presentation module204presents634the video stream on the mobile device115.

FIG. 7is a flow diagram illustrating one embodiment of a method700for switching between a video view and one of two different data views using a selection in a multi-user communication session. In one embodiment, the controller202receives702data indicating that a first participant, a second participant and a third participant125joined a multi-user communication session. The view presentation module204presents704a video stream of the multi-user communication session to a mobile device115associated with the third participant125. For example, the view presentation module204instructs the user interface engine210to generate graphical data for displaying the video stream. In one embodiment, the screen detection module206determines an occurrence of a detection trigger event. The controller202receives706a video frame image from the video stream that includes a first device associated with the first participant and a second device associated with the second participant. For example, the controller202receives a latest video frame image of the video stream from the camera103. The screen detection module206detects708a first data screen from the first device and a second data screen from the second device in the video frame image.

The controller202receives710data describing a selection of the first data screen performed on the mobile device115. For example, the controller202receives data describing a finger pressing in the center of the image of the first device to indicate that the third participant wants to view the first data view. The view switching module208switches712a view on the mobile device115from video view to a first data view that corresponds to the first data screen responsive to the selection. The view presentation module204presents714the first data stream on the mobile device115.