TACTILE OVERLAY FOR TOUCH SCREEN VIRTUAL GAME CONTROLLER COUPLED TO EXTERNAL DISPLAY

An electronic device, method, and computer program product combine the mobility and flexibility of presenting a virtual gamepad on a touch display with tactile feedback associated with physical gamepad controllers. The electronic device connects to an external display. A controller of the electronic device executes a game application to present, via the external display, a game screen. The controller detects a first display cover configured to overlay a touch display of the electronic device and having raised contour(s). The controller identifies a virtual gamepad configuration having virtual control(s) associated with the game application. The controller presents the virtual gamepad configuration on the touch display. The raised contour of the display cover is positioned to provide tactile feedback to guide interaction with the virtual control(s). Identification and/or orientation of the virtual gamepad configuration is based at least in part on detecting the display cover.

BACKGROUND

1. Technical Field

The present disclosure relates generally to communication devices that have a touch screen, and in particular to communication devices that can concurrently present different content on an internal touch screen and an external display.

2. Description of the Related Art

User communication devices may be sized for mobility, enabling users to engage in audio as well as video communication sessions in nearly any location. Within a small handheld form factor, communication devices present sensors and output devices that can be used as a traditional phone handset with a speaker positioned to be used as an earpiece and a microphone positioned to capture audio inputs as spoken by a user. Some communication devices such as smartphones have become multi-functional devices as the designs have increasingly incorporated a large number of capabilities. In an example, rather than being limited to manually actuated control keys and buttons, touch screens are configurable to present visual content and graphical controls in support of many types of applications in addition to user communication sessions.

In a related technology, game consoles are widely used, combining a large external display monitor for an immersive experience with a physical gamepad controller that is ergonomically configured for two-handed use while looking at the monitor. When unable to use a stationary game console, gamers increasingly use mobile gaming even though the small screen and lack of ergonomic game controls limits the gaming experience. To better approximate playing on a game console, mobile communication devices are being configured to present a virtual gamepad controller while presenting a game screen on a larger external display monitor.

DETAILED DESCRIPTION

According to aspects of the present disclosure, an electronic device, a method and a computer program product combine the mobility and flexibility of presenting a virtual gamepad on a touch display with passive tactile contours that provides touch feedback associated with physical gamepad controllers. The electronic device includes a touch display and a communication interface that enables the electronic device to connect to an external display. A first display cover is configured to overlay the touch display and has at least one raised contour. The electronic device includes a memory storing a first game application associated with a first game controller. A controller of the electronic device is communicatively connected to the external display, the touch display, and the memory. The controller executes the first game application to present, via the external display, a first game screen. The controller identifies a first virtual gamepad configuration including one or more virtual controls associated with the first game application. The controller presents the first virtual gamepad configuration on the touch display, where the at least one raised contour of the first display cover is positioned to provide tactile feedback to guide interaction with the at least one of the one or more virtual controls. In one or more embodiments, at least one of identification and orientation of the first virtual gamepad configuration is based, at least in part, on the controller detecting characteristic(s) of the first display cover via a sensor. In particular, the controller may detect identifying information for the first display cover that enables the controller to determine of one or more of: (i) presence of the first display cover; (ii) type of the first display cover associated with a corresponding virtual gamepad configuration; and (iii) orientation of the first display cover.

In the following detailed description of exemplary embodiments of the disclosure, specific exemplary embodiments in which the various aspects of the disclosure may be practiced are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, architectural, programmatic, mechanical, electrical, and other changes may be made without departing from the spirit or scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and equivalents thereof. Within the descriptions of the different views of the figures, similar elements are provided similar names and reference numerals as those of the previous figure(s). The specific numerals assigned to the elements are provided solely to aid in the description and are not meant to imply any limitations (structural or functional or otherwise) on the described embodiment. It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements.

As further described below, implementation of the functional features of the disclosure described herein is provided within processing devices and/or structures and can involve use of a combination of hardware, firmware, as well as several software-level constructs (e.g., program code and/or program instructions and/or pseudo-code) that execute to provide a specific utility for the device or a specific functional logic. The presented figures illustrate both hardware components and software and/or logic components.

Those of ordinary skill in the art will appreciate that the hardware components and basic configurations depicted in the figures may vary. The illustrative components are not intended to be exhaustive, but rather are representative to highlight essential components that are utilized to implement aspects of the described embodiments. For example, other devices/components may be used in addition to or in place of the hardware and/or firmware depicted. The depicted example is not meant to imply architectural or other limitations with respect to the presently described embodiments and/or the general invention. The description of the illustrative embodiments can be read in conjunction with the accompanying figures. Embodiments incorporating teachings of the present disclosure are shown and described with respect to the figures presented herein.

FIG.1is a functional block diagram of a communication environment100that includes an electronic device, which is referenced herein as communication device101and in which the features of the present disclosure are advantageously implemented. In particular, communication device101supports mobile gaming by presenting game screen102on external display103while presenting first virtual gamepad104on user interface105of touch display106. Communication device101includes communication subsystem107that enables communication device101to connect to external display103. Communication device101includes memory subsystem108storing game application109. Input/output (I/O) subsystem112of communication device101includes cover sensor117configured to detect presence, type, and orientation of display cover119. Controller110of communication device101is communicatively connected to external display103, touch display106, memory subsystem108, and cover sensor117.

Communication device101can be one of a host of different types of devices, including but not limited to, a mobile cellular phone, satellite phone, or smart phone, a laptop, a netbook, an ultra-book, a networked smartwatch or networked sports/exercise watch, and/or a tablet computing device or similar device that can include wireless communication functionality. As a device supporting wireless communication, communication device101can be utilized as, and also be referred to as, a system, device, subscriber unit, subscriber station, mobile station (MS), mobile, mobile device, remote station, remote terminal, user terminal, terminal, user agent, user device, a Session Initiation Protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), computer workstation, a handheld device having wireless connection capability, a computing device, or other processing devices connected to a wireless modem.

Referring now to the specific component makeup and the associated functionality of the presented components. In addition to communications subsystem107, memory subsystem108, and controller110, communication device101includes data storage subsystem111, and I/O subsystem112having input devices113aand output devices113b. To enable management by controller110, system interlink114communicatively connects controller110with communications subsystem107, memory subsystem108, data storage subsystem111, and input/output subsystem112. System interlink114represents internal components that facilitate internal communication by way of one or more shared or dedicated internal communication links, such as internal serial or parallel buses. As utilized herein, the term “communicatively coupled” means that information signals are transmissible through various interconnections, including wired and/or wireless links, between the components. The interconnections between the components can be direct interconnections that include conductive transmission media or may be indirect interconnections that include one or more intermediate electrical components. Although certain direct interconnections (i.e., system interlink114) are illustrated inFIG.1, it is to be understood that more, fewer, or different interconnections may be present in other embodiments.

In one or more embodiments, communications subsystem107may include one or more network interfaces115, such as local wireless communication module116and local wired communication module118, to communicatively couple communication device101via network cable120or wireless connection122to external networks124. Communication device101, via external networks124, may connect to network storage devices125that store computer data and to network server devices126that facilitate access to network storage device125. Network server devices126may have identical or similar components and functionality as described above for communication device101. Communication device101may communicate with second communication devices127via external network124or via communication networks132that are supported by core networks130. Network interface(s)115may include a network interface controller (NIC) and support one or more network communication protocols. External network124can be a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), or a wide area network (WAN). For example, network cable120and wireless connection122can be an Ethernet connection/cable.

In one or more embodiments, communications subsystem107may include additional functionality for communicating, using a cellular connection, with network node(s)134of external communication system136and for communicating, using a wireless connection, with wireless access point138or local wireless devices139of local communication system140. Communications subsystem107includes antenna subsystem144. Communications subsystem107includes radio frequency (RF) front end146and communication module148having baseband processor150. RF front end146includes transceiver(s)152, which includes transmitter(s)154and receiver(s)156. RF front end146further includes modem(s)158. Baseband processor150of communication module148communicates with controller110and RF front end146. Baseband processor150operates in a baseband frequency range to encode data for transmission and decode received data, according to a communication protocol. Modem(s)158modulates baseband encoded data from communication module148onto a carrier signal to provide a transmit signal that is amplified by transmitter(s)154. Modem(s)158demodulates each signal received using antenna subsystem144from external communication system136or local communication system140. The received signal is amplified and filtered by receiver(s)156, which demodulates received encoded data from a received carrier signal.

In one or more embodiments, controller110, via communications subsystem107, performs multiple types of cellular over-the-air (OTA) or wireless communication with local communication system140. Communications subsystem107can communicate via an OTA connection160with local wireless devices139. In an example, OTA connection160is a Bluetooth connection, or other personal access network (PAN) connection. In one or more embodiments, communications subsystem107communicates with one or more locally networked devices via a wireless local area network (WLAN) link162supported by access point138. In one or more embodiments, access point138supports communication using one or more IEEE 802.11 WLAN protocols. Access point138is connected to communication networks132via a cellular or wired connection. In one or more embodiments, communications subsystem107receives downlink channels164from GPS satellites166to obtain geospatial location information. Communications subsystem107can communicate via an over-the-air (OTA) cellular connection168with network node(s)134.

Controller110includes processor subsystem170, which includes one or more central processing units (CPUs), depicted as data processor171. Processor subsystem170can include one or more digital signal processors172that can be integrated with data processor171. Processor subsystem170can include other processors that are communicatively coupled to data processor171, such as baseband processors150of communication module148. In one or more embodiments that are not depicted, controller110can further include distributed processing and control components that are external to housing173or grouped with other components, such as I/O subsystem112. Data processor171is communicatively coupled, via system interlink114, to memory subsystem108. In one or more embodiments, data processor171is communicatively coupled via system interlink114to communications subsystem107, data storage subsystem111and I/O subsystem112. Controller110manages, and in some instances directly controls, the various functions and/or operations of communication device101. These functions and/or operations include, but are not limited to including, application data processing, communication with second communication devices, navigation tasks, image processing, and signal processing. In one or more alternate embodiments, communication device101may use hardware component equivalents for application data processing and signal processing. For example, communication device101may use special purpose hardware, dedicated processors, general purpose computers, microprocessor-based computers, micro-controllers, optical computers, analog computers, dedicated processors and/or dedicated hard-wired logic.

Memory subsystem108stores program code174for execution by processor subsystem170to provide the functionality described herein. Program code174includes applications such as game applications109, virtual gamepad application175, display cover detection application176, and other applications178. In one or more embodiments, several of the described aspects of the present disclosure are provided via executable program code of applications executed by controller110. In one or more embodiments, program code174may be integrated into a distinct chipset or hardware module as firmware that operates separately from executable program code. Portions of program code174may be incorporated into different hardware components that operate in a distributed or collaborative manner. Implementation of program code174may use any known mechanism or process for doing so using integrated hardware and/or software, as known by those skilled in the art. Memory subsystem108further includes operating system (OS)179, firmware interface180, such as basic input/output system (BIOS) or Uniform Extensible Firmware Interface (UEFI), and firmware181.

Memory subsystem108includes computer data182such as virtual gamepad configuration data183used by game application109to interface with virtual gamepad104, with virtual gamepad application176to identify virtual gamepad104that is compatible with game applications109, and with display cover detect application176that matches virtual gamepad104with display cover119. Computer data182may incorporate “data” that originated as raw, real-world “analog” information that consists of basic facts and figures. Computer data182includes different forms of data, such as numerical data, images, coding, notes, and financial data. The computer data may originate at communication device101or be retrieved by communication device101. Communication device101may store, modify, present, or transmit computer data182. Computer data may be organized in one of a number of different data structures. Common examples of computer data182include video, graphics, text, and images as discussed herein. Computer data182can also be in other forms of flat files, databases, and other data structures.

I/O subsystem112includes input devices113aand output devices113b. In addition to cover sensor117, input devices113amay include microphone186, image capturing devices187, and manual input devices188(e.g., keys and buttons). Output devices113bmay include audio output devices189in addition to external display103and touch display106. I/O subsystem112may include haptic or tactile output devices such as vibration device193that oscillates a mass such as battery194to create vibratory alerts. Touch display106provides user interface105having visual display layer192aas input device113abelow touch screen192bas output device113b.

Data storage subsystem111of communication device101includes data storage device(s)195. Controller110is communicatively connected, via system interlink114, to data storage device(s)195. Data storage subsystem111provides program code174and computer data182stored on nonvolatile storage that is accessible by controller110. For example, data storage subsystem111can provide a selection of computer data182and applications, such as game applications109, virtual gamepad application175, display cover detect application176, and other applications178. These applications can be loaded into memory subsystem108for execution/processing by controller110. In one or more embodiments, data storage device(s)195can include hard disk drives (HDDs), optical disk drives, and/or solid-state drives (SSDs), etc. Data storage subsystem111of communication device101can include removable storage device(s) (RSD(s))196, which is received in RSD interface197. Controller110is communicatively connected to RSD196, via system interlink114and RSD interface197. In one or more embodiments, RSD196is a non-transitory computer program product or computer readable storage device. Controller110can access data storage device(s)195or RSD196to provision communication device101with program code, such as program code174for game applications109, virtual gamepad application175, display cover detection application176, and other applications178, and with computer data182.

In one or more embodiments, controller110executes a first game application of the one or more game applications109to present, via external display103, game screen102. Controller110identifies a first virtual gamepad configuration in virtual gamepad data183having two or more virtual controls198. Controller110presents the first virtual gamepad configuration as first virtual gamepad104on touch display106. Display cover119may be placed over touch display106to enhance a game play experience by providing tactile feedback so that user199may focus attention on game screen102presented on external display103. In particular, raised contours184, such as hole edges177, raised areas190a, or depressed areas190bof covering layer191of display cover119are positioned to provide passive tactile contours for touch feedback to guide interaction at virtual controls198while gazing at external display103.

FIG.2depicts communication environment100, which includes gaming network system201in communication with first and second gaming consoles202aand202band communication device101. First gaming console202ais provisioned by gaming network system201with first game application109aand second game application109b. First physical gamepad controller204ahas first physical game controls206a(e.g., joysticks, joypads, buttons, bumpers, triggers, etc.) in a first gamepad physical layout. User199uses first physical gamepad controller204ato interact with first game console202a. First game console202ais executing first game application109aand presenting first game screen102aon second external display103a. Similarly, second gaming console202bis provisioned by gaming network system201with first game application109aand second game application109b. Second physical gamepad controller204bhas second physical game controls206b(e.g., joysticks, joypads, buttons, bumpers, triggers, etc.) in a second gamepad physical layout. User199uses second physical gamepad controller204bto interact with game console202b. Second game console202bis executing first game application109aand presenting first game screen102aon third external display103b.

In one or more embodiments, gaming network system201supports communication device101for mobile gaming. Gaming network system201may have a distributed computing architecture such as a cloud service that includes more than one network storage device125and more than one network server device126(FIG.2). Gaming network system201stores and distributes first and second game applications109aand109b. Gaming network system201may store and distribute one or more virtual gamepads104and104a, such as to communication device101that is performing mobile gaming and presenting first game screen102aon external display103.

Being familiar with at least one of physical gamepad controllers204aand204b, user199may benefit from having a similar gaming experience when doing mobile gaming using communication device101to present virtual gamepad104on touch display106and game screen102on external display103. In an example, user199is familiar with first physical gamepad controller204aand selects to use corresponding first virtual gamepad104athat simulates the full functionality of first physical game controls206a. Display cover119includes raised contours184, such as hole edges177formed in covering layer191around holes185that are positioned to provide tactile feedback for virtual controls198that simulate first physical gamepad controller204a. Alternatively, user199may prefer second physical game controller204b, or the game being played dictates use of a different set of virtual controls198that are provided by second virtual gamepad104b. Second display cover119aincludes raised contours184, such as hole edges177around holes185formed in covering layer191that are positioned to provide tactile feedback for virtual controls198that simulate second physical gamepad controller204b. Each display cover119aand119binclude respective passive or active target identification components211aand211bthat indicate to cover sensor117of communication device101the presence, type, and orientation of corresponding display cover119aand119b. In an example, cover sensor117uses ultra-wideband (UWB) radio technology to detect identification components211aand211bthat are UWB devices.

UWB devices can use a very low energy level for short-range, high-bandwidth communications over a large portion of the radio spectrum. UWB has traditional applications in non-cooperative radar imaging. Most recent applications target sensor data collection, precise locating, and tracking. Ultra-wideband is a technology for transmitting information across a wide bandwidth (>500 MHZ). This allows for the transmission of a large amount of signal energy without interfering with conventional narrowband and carrier wave transmission in the same frequency band. Regulatory limits in many countries allow for this efficient use of radio bandwidth and enable high-data-rate personal area network (PAN) wireless connectivity, longer-range low-data-rate applications, and the transparent co-existence of radar and imaging systems with existing communications systems.

FIG.3Adepicts communication device101being held by user199and presenting first virtual gamepad104having virtual controls198a-198n.FIG.3Bdepicts communication device101presenting first virtual gamepad104a(FIG.3A) augmented by first display cover119athat corresponds to first virtual gamepad104a(FIG.3A). Each virtual control198a-198nis exposed within corresponding hole185a-185n.

FIG.4depicts a side cross sectional view of communication device101and display cover119having raised contour184that is hole edge177around hole185presented proximate to virtual control198presented by touch display106of communication device101.FIG.5depicts a side cross sectional view of communication device101and display cover119ahaving raised contour184provided by thicker cover layer the presents raised area190aaround thinner cover layer that presents depressed area190bover virtual control198presented by touch display106of communication device101.FIG.6depicts a side cross sectional view of communication device101and alternate second display cover119bhaving raised contour184provided by thinner cover layer that presents depressed area190baround thicker cover layer that presents raised area190aover virtual control198presented by touch display106of communication device101.

FIG.7Adepicts communication device101presenting first virtual gamepad104aon touch display106based on identifying orientation of first display cover119athat corresponds to first virtual gamepad104a. Communication device101monitors cover sensor117that detects presence, type, and orientation of target identification components211a.FIG.7Bdepicts communication device101presenting second virtual gamepad104bon touch display106based on identifying orientation of second display cover119bthat corresponds to second virtual gamepad104b. The orientation of presentation of second virtual gamepad104bis rotated 180° as compared to presentation of first virtual gamepad104a(FIG.7A), corresponding to orientation of second display cover119b.

FIG.8Adepicts communication device101presenting first virtual gamepad104aon touch display106prior to being augmented by second display cover119bthat corresponds to second virtual gamepad104b.FIG.8Bdepicts communication device101covered by second display cover119b(shown in phantom lines) and automatically transitions to presenting on touch display106second virtual gamepad104bbased on identifying the type and the orientation of second display cover119bthat corresponds to second virtual gamepad104b.

FIG.9is a flow diagram presenting method900of automatically identifying presence, type, and orientation of a display cover that augments a touch screen of an electronic device presenting a corresponding virtual gamepad, according to one or more embodiments. The description of method900(FIG.9) is provided with general reference to the specific components illustrated within the precedingFIGS.1,2,3A,3B,4-6,7A,7B,8A, and8B. Specific components referenced in method900(FIG.9) may be identical or similar to components of the same name used in describing precedingFIGS.1,2,3A,3B,4-6,7A,7B,8A, and8B. In one or more embodiments, controller110(FIG.1) configures communication device101(FIGS.1,3A,3B,4and5) to provide the described functionality of method900(FIG.9).

With reference toFIG.9, method900includes connecting, via a communication interface of an electronic device, to an external display (block902). Method900includes executing a first game application to present, via the external display, a first game screen (block904). Method900includes presenting a default virtual gamepad configuration on the touch display (block906). Method900includes monitoring a cover sensor configured to detect one or more of presence, type, and orientation of a first display cover configured to be overlaid on the touch display (block908). Method900includes detecting a first display cover configured to overlay a touch display of the electronic device and having at least one raised contour (block910). In one or more embodiments, method900includes identifying a first virtual gamepad configuration including one or more virtual controls associated with one or more of: (i) the first game application and (ii) a type of the first display cover of more than one display cover (block912). Method900includes presenting the first virtual gamepad configuration on the touch display in an orientation corresponding to the first display cover (block914). The at least one raised contour of the first display cover is positioned to provide tactile feedback to guide interaction with the at least one of the one or more virtual controls. Method900includes detecting an input to the touch screen in the presence of the first display cover (block916). Method900includes automatically transitioning to presenting a corresponding virtual gamepad configuration in response to detecting a change in presence, type, and/or orientation of a display cover on the electronic device (block918). Then method900ends.

In one or more embodiments, the first display cover is configured to one of: (i) transfer one of a manual force or a capacitive touch to the touch display. The first display cover includes a first section having a first thickness and a second section adjacent to the first section and having a second thickness that is thicker than the first thickness, a transition from the first section to the second section defining the at least one raised contour.

In one or more embodiments, the at least one raised contour of the first display cover corresponds to at least one gamepad manual control of a first physical configuration of a first game controller. In one or more embodiments, the first display cover includes at least one hole in a covering layer, presenting a corresponding exposed portion of the touch display and defining the at least one raised contour (i.e., hole edge) surrounding the at least one hole.

In one or more embodiments, method900includes monitoring the cover sensor to detect one lateral edge of the touch display that is generally horizontal and that is vertically higher than three other lateral edges of the touch display. Method900includes presenting the first virtual gamepad configuration on the touch screen oriented with an upper portion aligned with the one lateral edge.

In one or more embodiments, method900includes storing, in memory, a second game application associated with a second game controller. Method900includes executing the second game application to present, via the external display, a second game screen. Method900includes identifying a second virtual gamepad configuration of one or more virtual controls that corresponds to one or more gamepad manual controls in a second physical configuration of a second game controller associated with the second game application. Method900includes presenting the second virtual gamepad configuration on the touch display to align with at least one raised second contour of a second display cover of a set of two or more display covers. The set of two or more display covers includes the first display cover and a second display cover configured to overlay the touch display and having the at least one raised second contour that corresponds to at least one manual control of the second physical configuration of the second game controller.

In one or more particular embodiments, method900includes monitoring the cover sensor to detect presence and orientation of an attached one of the first display cover or the second display cover overlaid on the touch display. Method900includes presenting the first virtual gamepad configuration on the touch screen at least in part in response to detecting, via the cover sensor, the presence of the first display cover overlaid on the touch display. Method900includes presenting the second virtual gamepad configuration on the touch screen at least in part in response to detecting, via the cover sensor, the presence of the second display cover overlaid on the touch display.

As will be appreciated by one skilled in the art, embodiments of the present innovation may be embodied as a system, device, and/or method. Accordingly, embodiments of the present innovation may take the form of an entirely hardware embodiment or an embodiment combining software and hardware embodiments that may all generally be referred to herein as a “circuit,” “module” or “system.”

While the innovation has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted for elements thereof without departing from the scope of the innovation. In addition, many modifications may be made to adapt a particular system, device, or component thereof to the teachings of the innovation without departing from the essential scope thereof. Therefore, it is intended that the innovation not be limited to the particular embodiments disclosed for carrying out this innovation, but that the innovation will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another.