System and method for touch-based communications

Disclosed is a system and method for an interactive communication experience on mobile devices. In general, the present disclosure discusses dynamically manipulating or modifying graphic user representations during an electronic communication. The modification or manipulation of these graphic user representations enables users to convey nuances of mood and feelings rather than being confined to conveying them through conventional communications, including text, images, video, or selecting an appropriate emoticon or avatar from a palette of predetermined emoticons or avatars.

FIELD

The present disclosure relates generally to customizable communications within messaging applications, and more particularly to dynamically manipulating graphic user representations during electronic communications in order to convey information to users of mobile communication devices.

RELATED ART

Mobile communication devices can provide users with various services such as voice calling, input/output services, and data storage. Mobile devices are also equipped with various complicated functions including capturing and/or modifying photos, audio and video, playing music and video files, providing game programs, receiving broadcast programs and providing wireless Internet services. Thus, mobile devices have evolved into multi-media players.

Mobile devices generally use auditory and visual cues to alert users when incoming messages are received. Mobile devices also can use vibratory alerts to alert users of incoming messages. Such alerts or cues are capable of simply providing users with notifications of incoming content.

SUMMARY

The present disclosure addresses failings in the art by providing a system and method for customizing or personalizing auditory, visual and vibratory alerts associated with user communications. That is, the present disclosure describes additional or new sensory modalities that deliver information to users of mobile devices in a personalized manner. Such sensory modalities enhance the reality of a user's experience in a virtual world. With the advent and mainstream usage of the touch screen interfaces on mobile devices, spatial senses affiliated with receiving electronic communications are unified. That is, touch capabilities combine the way a user navigates through the virtual world and communicates with other users by enabling a complete and personalized communication experience for users by combing sight, sound and touch with conventional communication protocol.

Generally, the present disclosure describes systems and methods for dynamically manipulating graphic user representations during an electronic communication, such as an instant message chat, SMS or other dialogue where users are instantaneously communicating over a network. Graphical user representations can be emoticons or avatars, but are not limited to such, as they can also be other types of images, graphics, icons, animations, and the like. The manipulation of these graphic user representations enables users to convey nuances of mood and feelings rather than being confined to conveying them through conventional communications.

In accordance with one or more embodiments, a method is disclosed which includes facilitating, over a network via a server computing device, visible display of a first graphical user representation associated with a first user of a first computing device and a second graphical user representation associated with a second user of a second computing device, the first and second graphical user representations being visibly displayed on the first and second computing devices in accordance with an instant messaging session between the first user and the second user; receiving, at the server computing device, an event signal from the first computing device, the event signal corresponding to input at the first computing device by the first user, the input comprising data indicative of the first user's interactions with the graphical user representation of the second user displayed on the first computing device during the instant messaging session; determining, via the server, an event based upon the interaction data within the event signal; generating, via the server, an event message for transmittal to the second computing device based upon the determined event, the event message comprising an action to be performed on the second computing device during the instant messaging session; and transmitting, from the server over the network, the event message to the second computing device for execution of the action during the instant messaging session.

In accordance with one or more embodiments, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium tangibly storing thereon, or having tangibly encoded thereon, computer readable instructions that when executed cause at least one processor to dynamically manipulate graphic user representations during an electronic communication.

In accordance with one or more embodiments, a system is provided that comprises one or more computing devices configured to provide functionality in accordance with such embodiments. In accordance with one or more embodiments, functionality is embodied in steps of a method performed by at least one computing device. In accordance with one or more embodiments, program code to implement functionality in accordance with one or more such embodiments is embodied in, by and/or on a computer-readable medium.

DESCRIPTION OF EMBODIMENTS

These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implement the functions/acts specified in the block diagrams or operational block or blocks.

For the purposes of this disclosure the term “server” should be understood to refer to a service point which provides processing, database, and communication facilities. By way of example, and not limitation, the term “server” can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and application software that support the services provided by the server. Servers may vary widely in configuration or capabilities, but generally a server may include one or more central processing units and memory. A server may also include one or more mass storage devices, one or more power supplies, one or more wired or wireless network interfaces, one or more input/output interfaces, or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

As used herein, the terms “text message” or “text messaging” refer to Short Message Service (SMS) messaging, as well as a variety of other limited size messaging protocols, including, but not limited to Instant Messaging (IM), Multimedia Messaging Service (MMS), or an Enhanced Message Service (EMS) messaging protocols, among other known or to be known communication protocols.

For purposes of this disclosure, an electronic computing device, electronic device or computing device (also referred to as a client device, user device or mobile device) may vary in terms of capabilities or features. Claimed subject matter is intended to cover a wide range of potential variations. For example, a cell phone may include a numeric keypad or a display of limited functionality, such as a monochrome liquid crystal display (LCD) for displaying text. In contrast, however, as another example, a web-enabled client device may include one or more physical or virtual keyboards, mass storage, one or more accelerometers, one or more gyroscopes, global positioning system (GPS) or other location-identifying type capability, or a display with a high degree of functionality, such as a touch-sensitive color 2D or 3D display, for example.

An electronic device may include or may execute a variety of operating systems, including a personal computer operating system, such as a WINDOWS®, iOS® or LINUX®, or a mobile operating system, such as iOS®, ANDROID®, or WINDOWS MOBILE®, or the like. An electronic device may include or may execute a variety of possible applications, such as a client software application enabling communication with other devices, such as communicating one or more messages, such as via email, short message service (SMS), or multimedia message service (MMS), including via a network, such as a social network, to provide only a few possible examples. An electronic device may also include or execute an application to communicate content, such as, for example, textual content, multimedia content, or the like. An electronic device may also include or execute an application to perform a variety of possible tasks, such as chatting, texting, browsing, searching, playing various forms of content, including locally stored or streamed video, or games (such as fantasy sports leagues). The foregoing is provided to illustrate that claimed subject matter is intended to include a wide range of possible features or capabilities.

According to some exemplary embodiments, the electronic device used herein is a touch sensor device, referred to as a touch device. A touch device is a device that typically includes a sensing region that uses capacitive, resistive, inductive, optical, acoustic or other technology to determine the presence, input (or depression), proximity, location and/or motion of one or more fingers, styli, pointers, and/or other objects. The touch device can be operated via input with one or more fingers styli, pointers and/or other objects, and can be used to provide an input to the electronic system, such as a desktop, tablet, notebook computer and smartphone, as well as kiosks and other terminals. As understood in the art, the touch device receives input not only when a user's finger(s) contacts the display screen of the touch device, but also when the user's finger(s) or other object(s) is within a detected proximity to the display screen of the touch device. Thus, the sensing region of the touch device can function as a cursor control/pointing device, selection device, scrolling device, graphics/character/handwriting input device, menu navigation device, gaming input device, button input device keyboard and/or other input device.

Although the embodiments discussed herein are described with reference to a touch device, other embodiments exist where the device is a computing device comprises, or is coupled to, a display screen where inputs are registered via a pointer (via a mouse), keyboard entry, or other inputs generally understood to register commands on a traditional computing device.

In addition, as discussed herein, exemplary embodiments occur within a GUI displayed on a touch device. However, it should be understood that any array of electronic devices can be used. Such devices, referred to as a client (or user) device may include a computing device capable of sending or receiving signals, such as via a wired or a wireless network. A client device may, for example, include a desktop computer or a portable device, such as a cellular telephone, a smart phone, a display pager, a radio frequency (RF) device, an infrared (IR) device, a Personal Digital Assistant (PDA), a handheld computer, a tablet computer, a laptop computer, a set top box, a wearable computer, an integrated device combining various features, such as features of the forgoing devices, or the like.

In various embodiments, the touch device provides a GUI for controlling a visual computing environment that represents programs, files, and options with graphical images, such as icons, menus and dialog boxes on the display screen of the touch device. Graphical items defined within the GUI can provide software routines which are handled by the GUI. Therefore, the GUI can report and act upon a user's actions respective of the graphical items. A GUI is a window, all of or a defined area of a display that contains distinguishable text, graphics, video, audio and other information for output.

The touch device can employ a touch screen interface, as discussed inFIGS. 3-4below. The touch screen interface combines the functionality of the GUI displayed on the display screen of with the touch device with the touch device's capabilities to recognize a sensed input. With most sensing technologies (e.g. capacitive, resistive, and inductive), a touch sensor is stacked with the display screen and sensor elements (e.g. electrodes) are located above, below, or within the display screen elements. Other technologies (e.g. surface acoustic wave and optical) may position the sensor elements elsewhere, but at least part of the sensing region overlaps with the display screen. The resulting combination is usually referred to together as a “touch screen.” A touch screen can provide a multi-purpose interface that can function both as a display and as an input device. Furthermore, because virtual touch screen controls can replace some physical input controls, the touch screen can extend to areas of a device typically reserved for other input devices.

The preferred embodiments of the present disclosure will now be described with reference toFIGS. 1-12. The present disclosure provides systems and methods for an interactive communication experience on mobile devices. In general, the present disclosure discusses dynamically manipulating or modifying graphic user representations during an electronic communication, such as an instant message chat, SMS or other dialogue where users are instantaneously communicating over a network. Graphical user representations can be emoticons or avatars, but are not limited to such, as they can also be other types of images, graphics, icons, animations, and the like. Additionally, the implementation and interaction with graphical user representations is not limited to the representations as a whole, as some embodiments exist where manipulation and interaction of graphical user representations involves a portion or portions of the representations. The modification or manipulation of these graphic user representations enables users to convey nuances of mood and feelings rather than being confined to conveying them through conventional communications, including text, images, video, or selecting an appropriate emoticon or avatar from a palette of predetermined emoticons or avatars.

One type of graphical user representation is an emoticon. Emoticons are small, typically simplistic, images representing a user's facial expression or current mood. For example, a “smiley face” as is an emoticon used to signal that the writer of the message is smiling or happy. In another example, a “sad face” is an emoticon conventionally used to signal that the writer of the message is unhappy or displeased. These emoticons therefore enable the recipient of a message to better understand the tenor of the writer's message.

Another type of graphical user representation, which is used in an exemplary but non-limiting manner in the present disclosure, is an avatar. An avatar is frequently used on the Internet or in gaming to represent a particular user. An avatar may include a picture or other images associated with a user, or an animated character in the context of an application. An application can be a computer-implemented game, a productivity application (such as a calendar or word processor), a communications application (such as email or chat), a social networking application, and any other suitable application. The application may be a web-based or network application implemented in connection with a web browser, or a stand-alone application hosted on a client device (or server). For example, an application can be an instant messaging chat application natively supported by a mobile device. In particular embodiments, an avatar may be a graphic icon representing a user in a computer user interface provided by an application. In some embodiments, an avatar may be a photographic picture representing a user.

Typically, graphical user representations, such as emoticons and avatars, are preselected from a predetermined set or palette of prefabricated images, graphics or icons. Because the user is confined to select his or her emoticon or avatar from a finite number of possible emoticons or avatars, the user is unable to convey personalized/customized nuances in mood or feeling. The present disclosure remedies these shortcomings by enabling a user to manipulate and/or tailor a graphical user representation in order to appropriately convey desired information.

By way of background, online services may provide users with the ability to send and receive instant messages, such as, for example, SMS messages. Instant messages are private online conversations between two or more people who have access to an instant messaging service, such as those who have installed communications software necessary to access and use the instant messaging service, and/or those who each generally have access to information reflecting the online status of other users. An instant messaging sender may send self-expressed items to a recipient. Such self-expressed items can include emoticons and avatars. However, there is always an emotional disconnect. That is, one person can chat (i.e., talk or text) with another person; however, there remains an emotional barrier between each user.

The present disclosure discusses the use of touch screens and mobile hardware to facilitate stronger communications and connections between users. Touch and mobile devices, as discussed below in relation toFIGS. 3-4, include hardware accelerometers, gyroscopes, vibrations and other feedback systems. These feedback systems, such as those incorporating haptic technology, can be utilized to make physical connections, such as those synonymous with shaking someone's hand or hugging your friend. Thus, based on these capabilities, graphical user representations, such as emoticons and avatars, can be used to convey mood and emotion. That is, the user representations can be dynamically manipulated, e.g., by squeezing, rotating, tapping, pinching, distorting, coloring, and the like, to convey specific information related to a mood, emotion or particular statement. Additionally, user representations can also be manipulated by shaking, tilting, spinning, and the like, the user's mobile device. This enables a user to customize or tailor an existing graphical user representation to better reflect the user's current mood or emotion, and even convey a message comprising such information related to a mood, emotion or statement.

In a non-limiting exemplary example in accordance with the examples illustrated inFIGS. 9-11, a user, Bobbi, is having a conversation with another user, Andreas. The conversation is a text conversation occurring over an instant messaging platform, such as, for example, iMessage™ on iOS® via each user's iPhone®, or in another example, over Yahoo! Messenger® on a user's mobile device. It should be understood that an instant messaging session can be supported by application programs native to users' devices, and in some embodiments, web-enabled application programs. Also, it should be understood, as currently provided by messaging applications, avatars, emoticons, or other images are displayed within the chat user interface, as discussed below and illustrated inFIGS. 9-11.

In the conversation, Bobbi has just learned that Andreas is not feeling well. In an attempt to cheer up Andreas, Bobbi can manipulate a displayed avatar to create an expression to be communicated to Andreas. For example, Bobbi can place her thumb on her touch screen in relation to Andreas' displayed avatar, and wiggle her thumb. In some embodiments, this can create a “tickle” affect. Thus, as a result, on Andreas' phone, her displayed avatar will shake slightly on the screen. Additionally, Andreas' phone may also shake slightly as a result of the “tickle” creating a haptic effect. This effect can occur simultaneously with the animation or according to another sequence under the directive of preconfigured or selected instructions by the system, user or combination thereof. In some embodiments, a notification, such as an emoticon notice, can appear on Andreas's device with an icon that says “you have been tickled.” In some embodiments, a notification or acknowledgement may also appear on Bobbi's phone stating “you have tickled Andreas.” Additionally, in some alternative embodiments, instead of interacting with Andreas' avatar, Bobbi can type the words “tickle” which can trigger the “tickle” effect discussed above. Therefore, instead of Bobbi being limited to simply sending a message implying a joke or request for Andreas to feel better, or even selecting a conventional static emoticon or avatar from a predefined library that resembles a similar connotation, Bobbi can personalize an expression and realize such expression though direct interaction with her mobile device.

It should be understood from the below discussion that the input and output from user interactions with a touch screen and avatar are not limited to those discussed in the examples herein, as the examples are non-limiting and merely discussed to portray embodiments and capabilities of the present disclosure. For example, users can interact with their mobile device's touch screen interface in all known and to be known manners, as discussed below.

FIG. 1shows components of a general environment in which the systems and methods discussed herein may be practiced. Not all the components may be required to practice the disclosure, and variations in the arrangement and type of the components may be made without departing from the spirit or scope of the disclosure. As shown, system100ofFIG. 1includes local area networks (“LANs”)/wide area networks (“WANs”)—network105, wireless network110, mobile devices (client devices)102-104and client device101.FIG. 1additionally includes a variety of servers, such as content server106, messaging server107and application (or “App”) server108.

One embodiment of mobile devices102-103is described in more detail below. Generally, however, mobile devices102-104may include virtually any portable computing device capable of receiving and sending a message over a network, such as network105, wireless network110, or the like. Mobile devices102-104may also be described generally as client devices that are configured to be portable. Thus, mobile devices102-104may include virtually any portable computing device capable of connecting to another computing device and receiving information. Such devices include portable devices such as, cellular telephones, smart phones, display pagers, radio frequency (RF) devices, infrared (IR) devices, Personal Digital Assistants (PDAs), handheld computers, laptop computers, wearable computers, tablet computers, integrated devices combining one or more of the preceding devices, and the like. As such, mobile devices102-104typically range widely in terms of capabilities and features. For example, a cell phone may have a numeric keypad and a few lines of monochrome LCD display on which only text may be displayed. In another example, a web-enabled mobile device may have a touch sensitive screen, a stylus, and several lines of color LCD display in which both text and graphics may be displayed.

A web-enabled mobile device may include a browser application that is configured to receive and to send web pages, web-based messages, and the like. The browser application may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web based language, including a wireless application protocol messages (WAP), and the like. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), and the like, to display and send a message.

Mobile devices102-104also may include at least one client application that is configured to receive content from another computing device. The client application may include a capability to provide and receive textual content, graphical content, audio content, and the like. The client application may further provide information that identifies itself, including a type, capability, name, and the like. In one embodiment, mobile devices102-104may uniquely identify themselves through any of a variety of mechanisms, including a phone number, Mobile Identification Number (MIN), an electronic serial number (ESN), or other mobile device identifier.

Mobile devices102-104may also be configured to communicate a message, such as through Short Message Service (SMS), Multimedia Message Service (MMS), instant messaging (IM), internet relay chat (IRC), Mardam-Bey's IRC (mIRC®), Jabber®, email, or the like, between another computing device or server. However, the present disclosure is not limited to these messaging protocols, and virtually any other messaging protocol may be employed.

Mobile devices102-104may be configured further to include a client application that enables the end-user to log into an end-user account that may be managed by another computing device, such as content server106, messaging server107and/or application server108. Such end-user account, for example, may be configured to enable the end-user to receive emails, send/receive IM messages, SMS messages, access selected web pages, or the like. However, the disclosure is not constrained, and in one embodiment, the end-user might not need to log into an account to send/receive messages.

In some embodiments, mobile devices102-104may also communicate with non-mobile client devices, such as client device101, or the like. In one embodiment, such communications may include sending and/or receiving messages, share photographs, audio clips, video clips, or any of a variety of other forms of communications. Client device101may include virtually any computing device capable of communicating over a network to send and receive information. The set of such devices may include devices that typically connect using a wired or wireless communications medium such as personal computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, or the like. Thus, client device101may also have differing capabilities for displaying navigable views of information.

Wireless network110is configured to couple mobile devices102-104and its components with network105. Wireless network110may include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, and the like, to provide an infrastructure-oriented connection for mobile devices102-104. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, and the like.

Wireless network110may further include an autonomous system of terminals, gateways, routers, and the like connected by wireless radio links, and the like. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network110may change rapidly. Wireless network110may further employ a plurality of access technologies including 2nd (2G), 3rd (3G), and/or 4th (4G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, and the like. Access technologies such as 2G, 3G, 4G and future access networks may enable wide area coverage for mobile devices, such as mobile devices102-104with various degrees of mobility. For example, wireless network110may enable a radio connection through a radio network access such as Global System for Mobil communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), and the like. In essence, wireless network110may include virtually any wireless communication mechanism by which information may travel between mobile device s102-104and another computing device, network, and the like.

Within the communications networks utilized or understood to be applicable to the present disclosure, such networks will employ various protocols that are used for communication over the network. Signal packets communicated via a network, such as a network of participating digital communication networks, may be compatible with or compliant with one or more protocols. Signaling formats or protocols employed may include, for example, TCP/IP, UDP, DECnet, NetBEUI, IPX, APPLETALK™, or the like. Versions of the Internet Protocol (IP) may include IPv4 or IPv6. The Internet refers to a decentralized global network of networks. The Internet includes local area networks (LANs), wide area networks (WANs), wireless networks, or long haul public networks that, for example, allow signal packets to be communicated between LANs. Signal packets may be communicated between nodes of a network, such as, for example, to one or more sites employing a local network address. A signal packet may, for example, be communicated over the Internet from a user site via an access node coupled to the Internet. Likewise, a signal packet may be forwarded via network nodes to a target site coupled to the network via a network access node, for example. A signal packet communicated via the Internet may, for example, be routed via a path of gateways, servers, etc. that may route the signal packet in accordance with a target address and availability of a network path to the target address.

In some embodiments, the disclosed networks110and/or105may comprise a content distribution network(s). A “content delivery network” or “content distribution network” (CDN) generally refers to a distributed content delivery system that comprises a collection of computers or computing devices linked by a network or networks. A CDN may employ software, systems, protocols or techniques to facilitate various services, such as storage, caching, communication of content, or streaming media or applications. Services may also make use of ancillary technologies including, but not limited to, “cloud computing,” distributed storage, DNS request handling, provisioning, signal monitoring and reporting, content targeting, personalization, or business intelligence. A CDN may also enable an entity to operate or manage another's site infrastructure, in whole or in part.

Accordingly, in some embodiments, the present disclosure may be utilized via a peer-to-peer network. In some embodiments, the peer-to-peer network may be embodied through a content distribution network. A peer-to-peer (or P2P) network may employ computing power or bandwidth of network participants in contrast with a network that may employ dedicated devices, such as dedicated servers, for example; however, some networks may employ both as well as other approaches. A P2P network may typically be used for coupling nodes via an ad hoc arrangement or configuration. A peer-to-peer network may employ some nodes capable of operating as both a “client” and a “server.”

According to some embodiments, the present disclosure may also be utilized within a social network. A social network refers generally to a network of individuals, such as acquaintances, friends, family, colleagues, or co-workers, coupled via a communications network or via a variety of sub-networks. Potentially, additional relationships may subsequently be formed as a result of social interaction via the communications network or sub-networks.

In some embodiments, multi-modal communications may occur between members of the social network. Individuals within one or more social networks may interact or communication with other members of a social network via a variety of devices. Multi-modal communication technologies refers to a set of technologies that permit interoperable communication across multiple devices or platforms, such as cell phones, smart phones, tablet computing devices, personal computers, televisions, set-top boxes, SMS/MMS, email, instant messenger clients, forums, social networking sites, or the like.

The content server106may include a device that includes a configuration to provide content via a network to another device. A content server106may, for example, host a site, such as a social networking site, or a personal user site (such as a blog, vlog, online dating site, and the like). A content server106may also host a variety of other sites, including, but not limited to business sites, educational sites, dictionary sites, encyclopedia sites, wikis, financial sites, government sites, and the like. Devices that may operate as content server106include personal computers desktop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, servers, and the like.

Messaging server107includes virtually any computing device that is configured to provide one or more messaging services. For example, messaging server107may provide Instant Messaging (IM) services, email services, VOIP services, SMS services, FTP services, or the like. Messaging server107may be implemented on a variety of computing devices including personal computers, desktop computers, multiprocessor systems, microprocessor-based devices, network PCs, servers, network appliances, or the like. Moreover, although messaging server107is illustrated as a single network device, the disclosure is not so limited. For example, one or more of messaging servers may also be implemented using a plurality of network devices to provide the various messaging services. According to some embodiments, messaging server107may be configured to receive messages from a sending device, such as client devices101-104, or the like, and provide the message, and/or a copy of the message to content server106for analysis. Furthermore, messaging server107may similarly receive a link to context, in one embodiment, from content server106, and provide the link or link information along with a response message to the original sending device. In another embodiment, messaging server107may forward the response message to content server106, and receive the response message, or a response message modified with the link information to the contextual message.

In an embodiment, users are able to access services provided by the content, message and application servers106,107and108. This may include in a non-limiting example, social networking services servers, SMS servers, IM servers, MMS servers, email servers, exchange servers, photo-sharing services servers, and travel services servers, via the network105using their various devices101-104. In some embodiments, applications, such as an instant messaging application, can be hosted by the application server108. Thus, the application server108can store various types of applications and application related information including user profile information in an application database, which is associated with the application server108.

Moreover, althoughFIG. 1illustrates content server106, messaging server107and application server108as single computing devices, respectively, the disclosure is not so limited. For example, one or more functions of content server106, messaging server107and/or application server108may be distributed across one or more distinct computing devices. Moreover, in one embodiment, content server106, messaging server107and application server108may be integrated into a single computing device, without departing from the scope of the present disclosure.

FIG. 2illustrates an exemplary architecture of a system200for managing, updating and communicating customizations and manipulations of graphical user representations in accordance with some embodiments of the present disclosure. The system200comprises a payment system240, a core system220, a display system230, a notification system240, a tool system250, and a user database associated with a content/service provider (UDB)260. Although the present system is illustrated as comprising a plurality of separate systems, the present disclosure is not so limited. Namely, greater or lesser number of systems can be deployed to perform the functions as described below. In fact, various systems as described below can be omitted if the functions supported by these systems are not deployed for a particular implementation. Additionally, although the user database (UDB)260is illustrated as a separate module, the present disclosure is not so limited. Namely, the user database (UDB)260can be deployed or distributed within one or more of the above systems.

In some embodiments, a payment system is optionally employed. For example, the payment system employs one or more billing servers242that allow users to purchase points in bulk. The purchased points can be used to purchase items such as special accessories or emotions, e.g., from an avatar store, for the users' online identities. Thus, a user may charge his or her credit card via the payment system to purchase “n” points that can be spent on avatar items. In one embodiment, the payment system240is extensible to support integration with third party billing, e.g., telephone/text charge billing and/or internet service billing.

Indeed, various monetization techniques or models may be used in connection with sponsored advertising, including advertising associated with user chats, communications and messages, and non-sponsored chat advertising, including graphical or display advertising. In an auction-based online advertising marketplace, advertisers may bid in connection with placement of advertisements, although many other factors may also be included in determining advertisement selection or ranking. Bids may be associated with amounts the advertisers pay for certain specified occurrences, such as for placed or clicked-on advertisements, for example. Advertiser payment for online advertising may be divided between parties including one or more publishers or publisher networks, and one or more marketplace facilitators or providers, potentially among other parties. Some models include guaranteed delivery advertising, in which advertisers may pay based on an agreement guaranteeing or providing some measure of assurance that the advertiser will receive a certain agreed upon amount of suitable advertising, and non-guaranteed delivery advertising, which may be individual serving opportunity-based or spot market-based. In various models, advertisers may pay based on any of various metrics associated with advertisement delivery or performance, or associated with measurement or approximation of a particular advertiser goal. For example, models can include, among other things, payment based on cost per impression or number of impressions, cost per click or number of clicks, cost per action for some specified action, cost per conversion or purchase, or cost based on some combination of metrics, which can include online or offline metrics.

In some embodiments, the core system220comprises one or more dedicated servers222for processing and handling avatar operations. For example, the core system220serves as the main entry points for users to browse and select items to purchase and wear for their avatars. The core system also comprises an avatar database224for holding avatar site data stored in a relational database and user data, stored in the User Database260. For example, as discussed below, the core system220can embody an event server and/or event database (or avatar database) which includes information related to animations of avatars and/or haptic effects related to interactions with avatars. Such embodiments are discussed in more detail below.

In some embodiments, the display system230comprises one or more generation servers234and one or more image servers232, where these servers are tasked with the generation and display of the avatars (or other graphical user representations). As discussed above, for ease of explanation, avatars will be discussed as an exemplary graphical user representation; however, it should be understood that any type of image, graphic, animation or file may be utilized in connection with the systems and methods discussed herein. For example, the display system230can either fetch avatar files from the storage system236, or generate them on the fly, caching the results on the storage system236. The storage system236may also keep pre-generated avatar files for other services (e.g., provided by a service provider) to obtain through a web interface.

In some embodiments, the notification system240comprises one or more real time servers242a-n, and at least one router244for routing avatar events. In operation, the avatar event router244in conjunction with messenger or mobile servers determines if an avatar user is logged into a real time messaging (or notification) service. If the query is positively answered, then the avatar event router244will pass notifications to the pertinent servers (242a-n) as required. To illustrate, if the user's avatar has experienced a change (e.g., a mood change, a clothing change, a background change, an accessory change and so on) and the user is logged into a real time messaging service, then the avatar change is sent via the notification system so that the change is presented in real-time. For example,FIG. 2illustrates messenger/mobile servers242nsending an avatar change notification to a mobile instant messenger client270, for displaying an updated avatar to a mobile device and/or to a messenger client280for displaying an updated avatar to an instant messenger application, e.g., running on a mobile device.

In some embodiments, the tool system250comprises one or more administration servers252for performing production, maintenance, and/or customer care functions. In one embodiment, the administration servers252may also allow third parties to submit content for approval, e.g., new representations (images and/or animation) of avatars, new accessories for avatars, new moods for avatars, new services for avatars, and so on. The tool system250allows the service provider to evaluate the submitted contents provided by third parties and to allow the service provider to update, modify or remove old contents. Finally, it should be noted that the system200is only exemplary, and can be modified to suit the requirement of a particular implementation.

In some embodiments, users are given limited free avatar customization abilities, and can then buy new animations, or combinations of hairstyles, clothes, accessories, and backgrounds for their avatar through a web-based storefront. In one embodiment, avatars are integrated into the Messenger client in an instant-message (IM) window and/or Friend List, e.g., a Yahoo!® IM window. Users may express themselves online with multiple moods and/or gestures. In some embodiments, users may customize their avatars by buying points that can then be spent on avatar animations, outfits, accessories, and backgrounds. Customization may take place through a web-based interface, and once complete, can be displayed through the Messenger, Mobile or Games client(s).

FIG. 2illustrates a block diagram depicting an exemplary embodiment of a real-time notification system in accordance with one or more aspects of the present disclosure. To illustrate, whenever a user creates a new avatar or modifies/manipulates/changes an existing avatar, the avatar core servers222, write the user's avatar to a service provider's unified database260, e.g., the Yahoo!® UDB. The avatar event router244, which is continuously listening for any changes to a user's record in the UDB260for avatar related information, picks up the avatar change notification. The avatar event router244sends the avatar information to the pertinent messenger and mobile servers242n, which then look up the user's messenger/mobile connection information and send an “avatar changed” event to the user himself and also to anyone who is interacting with the user, is logged into Messenger, and/or has the user in his/her buddy or contact list.

In some embodiments, when the Messenger client, e.g.,280receives an avatar change notification, the client caches the avatar key of the user and downloads the pertinent size (e.g., small, medium and large) of the avatars from the appropriate avatar platform where the user created his avatar. Once the small, medium and/or large avatars are downloaded on the client, the client shows the small avatar of the user in the messenger buddy list and the medium avatar of the user at the top of the Messenger client. It should be understood that the small, medium and/or large avatars may be local to a client device; therefore the avatar change notification can be handled locally by a client device. If the user is having a Messenger conversation with another user, the full avatar can be shown in the Messenger conversation (e.g., IM) window.

FIG. 3shows one embodiment of mobile device300that may be utilized within the present disclosure. Mobile device300may include many more or less components than those shown inFIG. 3. However, the components shown are sufficient to disclose an illustrative embodiment for implementing the present disclosure. Mobile device300may represent, for example, mobile devices102-104ofFIG. 1.

As shown in the figure, mobile device300includes a processing unit (CPU)322in communication with a mass memory330via a bus324. Mobile device300also includes a power supply326, one or more network interfaces350, an audio interface352, a display354, a keypad356, an illuminator358, an input/output interface360, a haptic interface362, and an optional global positioning systems (GPS) receiver364. Power supply326provides power to mobile device300. A rechargeable or non-rechargeable battery may be used to provide power. The power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements and/or recharges a battery.

Mobile device300may optionally communicate with a base station (not shown), or directly with another computing device. Network interface350includes circuitry for coupling mobile device300to one or more networks, and is constructed for use with one or more communication protocols and technologies including, but not limited to, global system for mobile communication (GSM), code division multiple access (CDMA), time division multiple access (TDMA), user datagram protocol (UDP), transmission control protocol/Internet protocol (TCP/IP), SMS, general packet radio service (GPRS), WAP, ultra wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), SIP/RTP, or any of a variety of other wireless communication protocols. Network interface350is sometimes known as a transceiver, transceiving device, or network interface card (NIC).

Audio interface352is arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface352may be coupled to a speaker and microphone (not shown) to enable telecommunication with others and/or generate an audio acknowledgement for some action. Display354may be a liquid crystal display (LCD), gas plasma, light emitting diode (LED), or any other type of display used with a computing device. Display354may also include a touch sensitive screen arranged to receive input from an object such as a stylus or a digit from a human hand.

Keypad356may comprise any input device arranged to receive input from a user. For example, keypad356may include a push button numeric dial, or a keyboard. Keypad356may also include command buttons that are associated with selecting and sending images. Illuminator358may provide a status indication and/or provide light. Illuminator358may remain active for specific periods of time or in response to events. For example, when illuminator358is active, it may backlight the buttons on keypad356and stay on while the client device is powered. Also, illuminator358may backlight these buttons in various patterns when particular actions are performed, such as dialing another client device. Illuminator358may also cause light sources positioned within a transparent or translucent case of the client device to illuminate in response to actions.

Mobile device300also comprises input/output interface360for communicating with external devices, such as a headset, or other input or output devices not shown inFIG. 3. Input/output interface360can utilize one or more communication technologies, such as USB, infrared, Bluetooth™, or the like. Haptic interface362is arranged to provide tactile feedback to a user of the client device. For example, the haptic interface may be employed to vibrate mobile device300in a particular way when the mobile device300receives a communication from another user.

Haptic technology refers to technology which interfaces to the user via the sense of touch by applying forces, vibrations and/or motions to the user. This mechanical stimulation may be used to assist in the creation of virtual objects (e.g., objects existing in a computing environment), for control of such virtual objects, and to enhance the control of machines and devices (e.g., mobile terminals). The implementation of haptic technology will be discussed in detail below with reference toFIGS. 4 and 6-8. Additionally, although the haptic effects related to vibrations are discussed herein, it should be understood that the effects of the present disclosure are not limited to vibratory pulses or effects, in that, auditory and/or visual effects can be output either alone or in combination with the vibrations (and/or animations).

Optional GPS transceiver364can determine the physical coordinates of mobile device300on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver364can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or the like, to further determine the physical location of mobile device300on the surface of the Earth. It is understood that under different conditions, GPS transceiver364can determine a physical location within millimeters for mobile device300; and in other cases, the determined physical location may be less precise, such as within a meter or significantly greater distances. In one embodiment, however, mobile device may through other components, provide other information that may be employed to determine a physical location of the device, including for example, a MAC address, IP address, or the like.

Mass memory330includes a RAM332, a ROM334, and other storage means. Mass memory330illustrates another example of computer storage media for storage of information such as computer readable instructions, data structures, program modules or other data. Mass memory330stores a basic input/output system (“BIOS”)340for controlling low-level operation of mobile device300. The mass memory also stores an operating system341for controlling the operation of mobile device300. It will be appreciated that this component may include a general purpose operating system such as a version of UNIX, or LINUX™, or a specialized client communication operating system such as Windows Mobile™, or the Symbian® operating system. The operating system may include, or interface with a Java virtual machine module that enables control of hardware components and/or operating system operations via Java application programs.

Memory330further includes one or more data stores, which can be utilized by mobile device300to store, among other things, applications342and/or other data. For example, data stores may be employed to store information that describes various capabilities of mobile device300. The information may then be provided to another device based on any of a variety of events, including being sent as part of a header during a communication, sent upon request, or the like. At least a portion of the capability information may also be stored on a disk drive or other storage medium (not shown) within mobile device300.

Applications342may include computer executable instructions which, when executed by mobile device300, transmit, receive, and/or otherwise process audio, video, images, and enable telecommunication with another user of another client device. Other examples of application programs include calendars, browsers, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so forth. Applications342may further include messaging client345that is configured to send, to receive, and/or to otherwise process messages using SMS, MMS, IM, email, VOIP, and/or any of a variety of other messaging communication protocols. Although a single messaging client345is illustrated it should be clear that multiple messaging clients may be employed. For example, one messaging client may be configured to manage SMS messages, where another messaging client manages IM messages, and yet another messaging client is configured to manage emails, or the like.

FIG. 4is a block diagram of an exemplary electronic device400(or system) that is coupled to a touch screen interface402for use in an exemplary embodiment of the present disclosure. Touch screen interface402can be implemented as part of a larger electronic system, or coupled to electronic device400using any suitable technique. For example, touch screen interface402can be communicably coupled to electronic device400through any type of channel or connection, including serial, parallel, 12C, SPI, PS/2, Universal Serial Bus (USB), Bluetooth, RF, IRDA, or any other type of wired or wireless connection to list several non-limiting examples. Similarly, the various elements of electronic device400(e.g. processing components, circuitry, memory, casing materials, physical hardware, and the like) can be implemented as part of an overall system, as part of touch screen interface402, or as a combination thereof.

The term “electronic device” is used to refer broadly to any type of device that communicates with a “touch screen interface.” As discussed above, the electronic device400can be a mobile device, and thus comprise any type of device or devices in which touch screen interface402can be implemented in or coupled to. As non-limiting examples, electronic device400can comprise any type of personal computer, portable computer, workstation, tablet, smartphone, personal digital assistant, video game player, communication device, media device, an input device, or a combination thereof. These examples are meant to be representative and broadly construed. Also, it should be understood that the terminology “electronic device” and “mobile device” are interchangeable with respect to their terms of capabilities or features.

For example, communications devices can include wired phones, wireless phones, and electronic messaging devices; input devices include touch sensors such as touch screens and touch pads, keypads, joysticks and mice, and remote controls; media devices recorders and players include televisions, music recorders and players, and set-top boxes such as cable descramblers and video recorders or players; and combination devices include cell phones with built-in cameras, PDAs that can double as electronic messaging systems or cell phones, and the like. In some embodiments, electronic device400can be itself a peripheral to a larger system, and communicates with another device (in addition to the touch screen interface402) using a suitable wired or wireless technique. Examples of peripherals include a remote control for a television, set-top box, or music system, a terminal on a wired network, and a media device capable of downloading media wireless from a separate source. Accordingly, the various embodiments of electronic device400may include any type of processor, memory, display, or other component as appropriate, and the elements of device400may communicate via a bus, network, or other wired or wireless interconnection as applicable. Additionally, electronic device400can be a host or a slave to touch screen interface402. The interactions involving one or more users and electronic device400can also take place on additional non-touch screen devices such as a mouse cursor and a traditional computer monitor.

The touch screen interface402includes a display screen404and a touch sensor device406, both of which are communicably coupled to processor408. Display screen404is any type of electronic display capable of displaying a visual interface to a human user, and can include any type of LED, CRT, LCD, plasma, or other display technology. Touch sensor device406is sensitive to some aspect of object motion of one or more input objects410such as fingers and styli in its sensing region.

It should be noted that although the various embodiments described herein refer to “touch sensor devices,” “proximity sensors,” or “touch pads,” these terms as used herein are used synonymously herein, and intended to encompass not only conventional touch sensor devices, but also a broad range of equivalent devices that are capable of detecting positional information about one or more fingers, pointers, styli and/or other objects. Such devices may include, without limitation, touch pads, touch tablets, biometric authentication devices, handwriting or character recognition devices, and the like. Thus, the interactions between one or more users and touch screen interface402could include a touch screen interface402with a touch sensor device406and one or more fingers, styli, other input objects412, or a combination thereof.

In touch screen interface402of electronic device400, processor408is coupled to touch sensor device406, display screen404and haptic interface414(as discussed above inFIG. 3in relation to item362). Generally, processor408receives electrical signals from touch sensor device406, processes the electrical signals, and communicates with display screen404. As discussed below, the processor408is operative to download, install and/or run application410which provides the capability to communication over a network. The application410is implemented, or executed, by the processor408through interactions with the touch sensor device406. For example, application410can be an instant messaging program utilized by the device400to communicate with other users. Also, the application410can be utilized to load, render and/or download graphical user representations. Processor408would also typically communicate with electronic device400, providing indications of input received on touch sensor device406and perhaps receiving information or instructions in turn. In exemplary embodiments, application410is an IM application enabling a user of the device400to interact with at least one other user, as discussed herein.

As discussed above, the haptic interface414is arranged to provide tactile feedback to a user of the client device. For example, the haptic interface414may be employed to vibrate mobile device400in a particular way when another user of a computing device is sending a communication, which is discussed below in relation toFIGS. 6-11. In some embodiments, and by way of a non-limiting example, processor408can receive an input signal, over a network from a server (or from the touch screen interface402), determine a source or characteristics of the event and select the control signal based on the determination, and output a control signal to the haptic interface414. The control signal is configured to cause the haptic interface414to output a haptic effect associated with the event. In some embodiments, the processor408will execute the control signal in accordance with instructions received from the server. For example, the haptic interface414will receive haptic code within control signals received over a network from a server or other device, and will output the haptic effect associated with the received control signal(s). Particular embodiments related to haptic effects will be discussed below in more detail.

FIG. 5is a flow chart of a process for manipulating or animating an avatar during an instant messaging session500in accordance with some embodiments of the present disclosure. As discussed above, a user can effectuate a communication to a recipient user which triggers the recipient user's avatar to react. For clarity, the process ofFIG. 5will be discussed in relation to an example. It should be understood that the steps of the process and the occurrences within the example are illustrative in nature, and not limiting, binding or exhaustive regarding the steps, users or iterations involved. For example, user501, Bobbi, has an established instant messaging session500with user503, Roman, where Bobbi501and Roman503are communicating over a network hosted by server/host505. In the example, Bobbi and Roman are dating, and Bobbi wants to send Roman a message stating that she hopes he feels better because she has just learned Roman is feeling ill. Both Bobbi and Roman have each other's avatar displayed on each of their respective devices. That is, within the chat interface, Bobbi's avatar is displayed on Roman's device, and Roman's avatar is displayed on Bobbi's device. (Depictions of the display of such avatars can be found inFIGS. 9-11, e.g.,1000aand1000binFIG. 10.) Each avatar is a graphical image that represents each user for self-expression in the instant message communications session500. In some embodiments, an avatar can support a predetermined number of animations corresponding to emotions/moods, e.g., normal (or straight face), smiling, happy, sad, and angry. However, additional moods and/or gestures known or to be known within a virtual landscape can also be realized. It should be understood that the length, frequency and type of animation related to an emotion/mood of an avatar can be predetermined (either set by the user, device and/or system), or based upon the received input effectuating the animation.

Turning to the example illustrating the process ofFIG. 5, instead of being constrained to sending a conventional text message stating “I hope you feel better” or inputting an emoticon representing the same, such as a “sad face,” Bobbi can interact with Roman's displayed avatar, which is displayed on her device501, and send Roman a message expressing her emotion. Step502. For example, Bobbi can provide input on her device respective of Roman's displayed avatar. For example, Bobbi can place her finger, e.g. index finger, thumb or some combination thereof, on Roman's displayed avatar. It should be understood that interaction with a displayed avatar can take many forms, including, but not limited to, tapping, swiping, pinching, wiggling, double tapping, and other known or to be known interactions with a touch screen interface.

Here, Bobbi has placed his finger (1001fromFIG. 10) on the avatar1000aand wiggled his finger, as illustrated in screen1004inFIG. 10. (In another example, Bobbi can “pinch” the avatar by depressing two fingers on Roman's displayed avatar and squeezing her two fingers together, as illustrated in screen908inFIG. 9). The interaction with the avatar triggers the generation of an event trigger, signal, or event message, to be sent over the network to Roman. Step504. The event message is transmitted from Bobbi's device to a host, or event server (associate with an event database). Step506. The event message is processed by the event server. Step508. That is, the event server receives the event message and determines which event the event message corresponds to by matching the event message to a specific event within an event library. That is, the information contained in the event message, which includes data associated with Bobbi's interaction with her touch screen interface, is identified (or parsed out) and compared against a library of events. This may be accomplished by using a database table, a list, or a file that associates one or more triggers with a type of animation for the avatar to identify a particular type of animation. Types of animation include, by way of example and not limited to, a smile, a wink, a frown, an expression with a tongue out, a shocked expression, a kiss, a hug, a yell, a big smile, a sleeping expression, a nodding expression, a sigh, a tickle, a sad expression, a cool expression, a laugh, a disappearance, a smell, dancing, or a negative expression. It should be understood that the animations applicable to the present disclosure include those that are known or to be known within the scope of animating, modifying, manipulating, distorting, and the like, images, graphics, videos and/or multimedia files. Also, the event library can be housed within the event server, event database, client device or some combination thereof. Thus, it should also be understood that animations can be predetermined by a user, a content provider, the system, or some combination thereof. In some embodiments, the information contained in the event message may not need to be compared against a library of event. In these embodiments, the information can be identified and utilized in creating a customized event message (or control signal) for transmission to the recipient's device.

In some embodiments, the event library is stored locally on the sender's device. In other embodiments, the event library is stored locally on the recipient's device. Native event libraries are useful for embodiments where the communications are occurring directly between devices, such as with infrared, Near Field Communications (NFC), when one device is acting as a proxy or hosting a local hotspot, or the like. In some embodiments, the event library corresponds to an event database that is affiliated with an event server. It should be understood that the event server can be a separate sever, or a server hosting the chat session, such as the messaging server107fromFIG. 1or within the core system220fromFIG. 2. Alternatively, the event server can be another server, such as the content server or application server, or a combination of applicable servers providing chat services or avatar resources.

In Step510, after the event is identified based on the transmitted event message from the sender's device (e.g., Bobbi's device), an event message is communicated to the recipient's device (e.g., Roman's device). The message contains signals or instructions to modify, manipulate or manage animations of Roman's avatar which is displayed on Roman's device503. Step512. The signals/instructions include information related to the type of animation, modification or manipulation, the duration of such, and where applicable, the frequency of such. Also, the scope of such animation, manipulation or modification is comprised within the message. In the above example, the avatar on Roman's device would be animated, as per the event message, to signify that Roman has been “hugged.” In some embodiments, in accordance with the “hug” event, Roman would receive a message that states “you have been hugged.” In some embodiments, Bobbi may also receive a message stating “you have hugged Roman.” An illustrating example of the process and example discussed above forFIG. 5is illustrated inFIGS. 9-11, specifically inFIG. 10. In some embodiments, as discussed inFIGS. 6-8, the “hug” can also trigger Roman's device to shake, thereby effectuating the haptic effects of the “hug.”

In an alternative embodiment, a sender can trigger an animation on a recipient's device by typing a message, whereby the message is parsed, and analyzed to identify an event trigger resulting in an event message, as discussed above. For example, the text of an instant message sent by the sender may trigger an animation of the recipient's avatar. Also, the text of an instant messages sent by the instant message recipient to the sender may trigger an animation of the sender's avatar. For example, the text of a message may include a character string “LOL,” which is an acronym that stands for “laughing out loud.” The character string “LOL” may trigger an animation in the sender avatar or the recipient avatar such that the sender avatar and/or the recipient avatar appear to be laughing.

Therefore, the ability to dynamically change the mood and/or gestures of the avatars provides a unique way to allow users to express their simulated mood and/or to express a simulated gesture. Using the dynamic moods of the avatars, users are afforded the ability to express themselves without resorting to the use of static imagery, or cumbersome functionality, such as web cameras where the users are allowing the other users to see them. For privacy reasons and/or resource reasons, avatars serve as simulated representations of the users that allow the users to express themselves freely. The dynamic nature of the avatars enhances the user's interactive experience while maintaining privacy.

FIG. 6is a flowchart depicting a method of using customized haptic effects to convey information to users of mobile communication devices, according to an embodiment of the present disclosure. As discussed above, a user can effectuate a communication to a recipient user which triggers the recipient user's avatar to react. In this embodiment, the reaction of the user's avatar causes the recipient user's device to produce a haptic event. Such event can be a vibration, audio alert, or a combination of the two. Additionally, in connection with the reaction produced from the embodiment inFIG. 5, the haptic event can occur simultaneously with the animation of the recipient's avatar, which will be discussed inFIG. 8.

As discussed herein, a mobile device, such as a smartphone, PDA or tablet computing device may be configured to allow a user to include haptic information or a haptic code in an outgoing communication signal, e.g., a text message. The encoding of a communication signal with haptic information may be based on an established scheme or protocol, and/or on a per-system basis. The haptic code is configured to cause a haptic effect to be output when the communication signal is delivered to another mobile device. In some embodiments, specific haptic code is associated with distinct haptic events, e.g., particular vibration patterns.

FIG. 6illustrates a process600for using customized haptic effects to convey information to users on their mobile devices. At step602, an input signal associated with an event is received, where the input signal comprises information related to a haptic event. For example, Bobbi provides input via the touch interface of her mobile device by tapping on Roman's displayed avatar. Bobbi's interaction corresponds to her desire to “hug” Roman. In step604, the input signal is transmitted to a server. The input signal, which comprises information identifying the haptic event “hug” is processed by the server. Step.606. That is, server extracts haptic code from the input signal. The haptic code provides information to the server identifying the specific input provided by Bobbi. In step608, the server identifies a haptic event based on the extracted haptic code. That is, for example, the information contained in the haptic code, which includes data associated with Bobbi's interaction with her touch screen interface, is compared against a library of haptic events. This may be accomplished by using a haptic library, or database table, a list, or a file that associates haptic code with a type of haptic event. As with the event server/database discussed above inFIG. 5, the haptic library can be located on a server, associated or independent database, or on one of the communicating devices. As noted above, specific haptic code can be associated with distinct haptic events, e.g., a particular vibration patterns in order to simulate the desired effect. In some embodiments, the haptic code may be configured according to a predetermined scheme or protocol that includes, for example, a table of haptic codes (some of which may be associated with one or more haptic events) versus control signals for rendering the corresponding haptic effects. In this way, a processor in the mobile device can look up the corresponding control signal from the table based on the extracted haptic code, and output the selected control signal to a haptic interface for rendering the desired haptic effect. Alternatively, the haptic code need not be compared against a library of codes/events, in that the haptic code can be utilized or implemented in generating an event control signal for outputting a haptic event, as discussed below. It should be understood that, according to some embodiments, haptic effects can be predetermined by a user, a content provider, the system, or some combination thereof.

Types of haptic events include, but are not limited to: vibrations, pulses, tones, auditory effects, and the like, whereby the vibrations, pulses, tones can vary force, length, duration, frequency, repetition and volume. For example, Bobbi intends to “hug” Roman. The haptic code extracted identifies the desired “hug”; therefore, the haptic effect identified in the haptic library relates to a particular vibration pattern—which for example can be two quickly repeated pulses (or two 0.3 second pulses within a 1 second time frame, for example). Haptic effects (along with associated control signals) may also be downloaded or transmitted from a remote source, such as a service provider, a network resource, a Web server, a remote handheld communication device or computer. Such downloaded or transmitted haptic effects can be further edited or modified. It is envisioned that third-party applications for handheld communication devices may enable users to purchase and download additional haptic effects as well as to edit and modify them. Additionally, users may be able to customize or modify such haptic effects. The haptic effects may also be provided in the form of a bundled package that also includes visual effects and audio effects which may or may not be configured to be synchronized with the haptic effects.

In step610, the server generates a control signal including the haptic effect, and transmits the control signal to another mobile device. For example, the control signal is transmitted to Roman's device. Upon reception at Roman's device, the haptic interface (or actuator) on Roman's device analyzes the control signal, and processes the haptic effect (or feedback). Step612. That is, the control signal is configured to cause the haptic interface to output the haptic effect. In some embodiments, the haptic effects can also be triggered by, or synchronized with, other occurrences, as discussed below inFIG. 8.

In an exemplary embodiment, some of these haptic codes may be associated with haptic effects that emulate expressions or behaviors, such as “laugh,” “giggle,” “tickle,” “hug,” “high-five,” “heartbeat,” “pet purring,” etc. This allows haptic effects to be transmitted and experienced, e.g., in an interactive conversation or a chat session, by pressing or manipulating displayed objects or portions on a touch screen interface. It should be understood, that the present disclosure is not limited to use on touch mobile devices, as the functionality discussed herein can be effectuated on any known or to be known device via input functionality/capabilities present on such devices.

By way of a non-limiting example, suppose that user Bobbi is engaged in a chat session with James via their respective mobile phones. In one embodiment, when Bobbi tells James a joke, instead of responding by typing text signifying laughter, such as “haha”, or inserting and sending an emoticon smiling within a message, James can respond by sending a “laugh” sensation to Bobbi, e.g., by pressing on the touch screen interface area respective of the Bobbi's displayed avatar, as discussed above. This causes a signal to be transmitted from James' phone to Bobbi's phone, and a corresponding haptic effect to be output to Bobbi's phone (and thereby experienced by Bobbi).

In accordance with alternative embodiments, James may also be able to include a haptic code in an outgoing message (which may also contain a video image such as a picture taken by his mobile phone, and/or a graphical feature such as an emoticon emulating a smiley face) to be transmitted to Bobbi. The haptic code causes a haptic effect to be output when the message is delivered to a remote device such as Bobbi's mobile phone. In accordance with one embodiment, the haptic effect may be correlated or synchronized with the displaying of a video image contained in the message. In accordance with another embodiment, the generation of the haptic effect based on the haptic code may be carried out in a manner similar to that described above with respect to the embodiments discussed herein.

Haptic effects can also be used to enhance and complement the information content communicated between mobile communication devices. In accordance with some embodiment, a plurality of handheld communication users may be engaged in a chat session via their mobile communication devices. As discussed herein, the users may each have a graphical user representation or avatar displayed on other mobile communication devices. Such avatars can also be haptically enabled, for example, whereby their expressions and/or behaviors are accompanied and enhanced by corresponding haptic effects.FIG. 7is a flowchart700depicting a method of carrying out a chat session using handheld communication devices, according to some embodiments of the present disclosure. In an embodiment, a mobile device receives an input signal associated with a chat message at step702. The mobile device displays an avatar associated with the chat message at step704. At step706, the handheld communication device provides a haptic effect associated with the chat message. Step706may include outputting a control signal to a haptic interface coupled to the handheld communication device, where the control signal is configured to cause the haptic interface to output the haptic effect. In accordance with one embodiment, the haptic effect may be correlated with an expression or behavior of the avatar, such as a laugh or giggle, a cry, a scream, dancing, a pet purring, or the like. The steps of processing the haptic effects are processes in a similar manner discussed above in relation toFIG. 6.

FIG. 8is a flow chart of a process for manipulating or animating an avatar during an instant message and providing an associated haptic effect in accordance with some embodiments of the present disclosure.FIG. 8illustrates an embodiment where the processes ofFIG. 5andFIG. 6are occurring in concert with one another, where a triggered event signal comprises information related to a desired avatar reaction and a corresponding haptic effect.

As discussed above, a user can effectuate a communication to a recipient user which triggers the recipient user's avatar to react. Additionally, the avatar's reaction can be coupled to, or connected with a haptic effect. From the examples used inFIGS. 5-6, user801, Bobbi, has an established instant messaging session800with user803, Roman. Step800. Bobbi and Roman are communicating over a network hosted by server/host805. As with the above example, Bobbi wants to send Roman a message stating that she hopes he feels better. Both Bobbi and Roman have each other's avatar displayed on each of their respective devices. That is, within the chat interface, Bobbi's avatar is displayed on Roman's device, and Roman's avatar is displayed on Bobbi's device. In step802, Bobbi can interact with Roman's displayed avatar and send Roman a message expressing her emotion. That is, Bobbi can provide input on her device respective of Roman's avatar. For example, Bobbi can place her finger, e.g. index finger, thumb or some combination thereof, on Roman's displayed avatar. It should be understood that interaction with a displayed avatar can take many forms, including, but not limited to, tapping, swiping, pinching, double tapping, and other known or to be known interactions with a touch screen interface.

In this example, Bobbi has “pinched” the avatar by depressing two fingers on Roman's displayed avatar and squeezing her two fingers together. The “pinch” triggers an event signal, or event message, to be sent over the network to Roman. Step804. The event signal is transmitted from Bobbi's device to a host, or event server (an associated event database). Step806. The event signal is processed by the event server (as discussed inFIG. 5). Step808.

Additionally, as discussed inFIG. 6, the event signal can also comprise information identifying the haptic event “hug”, which is also processed by the server805. Step809. That is, server extracts haptic code from the event signal. The haptic code provides information to the server identifying the specific input provided by Bobbi. In step810, the server identifies a haptic event based on the extracted haptic code. That is, for example, the information contained in the haptic code, which includes data associated with Bobbi's interaction with her touch screen interface, is compared against a haptic library of haptic events. As discussed above, this may be accomplished by using a database table, a list, or a file that associates haptic code with a type of haptic event.

In step811, after the event is identified in the event library and haptic library, the server generates a control signal which is transmitted to the recipient's device (e.g., Roman's device). It should be understood that the event library and haptic library discussed above can be within a single database or lookup table, or in separate, but connected databases or lookup tables. Additionally, in some embodiments, identifying the desired event or haptic effect can be based upon, sequentially identified or simultaneously determined in accordance with the other respective library.

The control signal contains signals or instructions to modify, manipulate or manage animations of Roman's avatar, and includes an associated the haptic effect. Upon reception at Roman's device, Step812, the avatar on Roman's device would be animated, as per the control signal, to signify that Roman has been “hugged.” Additionally, and in an exemplary embodiment, simultaneously, the haptic interface (or actuator) on Roman's device outputs the haptic effect. Step814. In some embodiments, the control signal comprises both the information for the animation and the information for the haptic effect. In some embodiments, more than one control signal may be transmitted respective of the information for the animation and the information for the haptic effect.

Alternative embodiments also exist relating to providing input, or generating an input signal on a mobile device. In addition to interacting with a touch screen interface, a user can also effectuate input by, for example, tilting the device at a predetermined angle, or shaking the device with a predetermined force, so as to register an ascertainable input and direction, which can be recognized as user input. The input can be based upon the direction, speed, velocity, acceleration, length of a tilt or shake, or some combination thereof, in addition to being based upon the angle of the tilt and/or force of such shaking of the device. Therefore, tilting and/or shaking the device can produce a user input (effectuating similar input as that of touching a touch screen interface) provided it is configured with acceleration and/or position and/or velocity sensors known in the art or later to become known.

To illustrate the embodiments and capabilities discussed above, specifically discussed in relation toFIGS. 5-8,FIG. 9-11depict various non-limiting implementations and examples of the present disclosure. It should be understood that the implementations and examples discussed forFIGS. 9-11are illustrative in nature, and not limiting, binding or exhaustive regarding the steps, users or iterations involved

FIG. 9depicts a touch-screen mobile device900, which through various implementations of screens902-908, animations and haptic effects can be realized as discussed above in relation toFIGS. 5-8. As shown in screen902, a user, via his/her finger (or styli)903, can interact with a user's avatar901. This interaction is an illustration of the input discussed inFIGS. 5-8. As a result of the interaction/input, screen904depicts one embodiment of a result of the interaction/input. As shown in relation to screen904, on the user's device who has produced the input, an acknowledgement/notification905is displayed. For example, upon depressing on another user's avatar, and triggering a “tickle”, the user receives an acknowledgement/notification905stating “you tickled Roman” next to their respective avatar903. (Roman being the other user communicating with the sending user, Bobbi, as from the above examples). As discussed in detail above, the “tickle” produces an animation on Roman's device. In some embodiments, the user receiving the animation (or haptic effect) will also realize the result. Here, in screen906, Roman's avatar901, which is displayed on his screen, is animated to realize the “tickle.” For example, Roman's avatar907may wiggle slightly and the device900may vibrate in accordance of the avatar's animation.

Screen908depicts an example of a type of interaction with an avatar via a touch screen interface of device900. Here, a user, instead of merely tapping on the screen, can also “pinch” the avatar901, as shown. (This is discussed in relation to the example/embodiment discussed above in relation toFIGS. 5 and 8.) In some embodiments, an avatar901will shrink in line with the “pinch” motion, and then “pop” back to normal size. This interaction/input, along with other types of interactions are known and understood in the art as capabilities and options for interacting with a touch screen interface/device. Therefore, it should be understood that known or to be known methods of interacting with a touch screen interface/device are applicable and able to be implemented in accordance with the present disclosure.

As discussed above,FIG. 10depicts embodiments/examples1002,1004and1006for “hugging” another user in accordance with functionality and capabilities discussed herein. The depictions inFIG. 10are discussed above in detail related toFIGS. 5 and 8.

FIG. 11depicts a touch-screen mobile device executing an instant messaging (IM) application, as discussed above. The user interface display1100shows an IM conversation between Bobbi and Roman. Each user's avatar1104and1106is displayed alongside their respective chat messages1102and1103. As shown by way of example in the particular implementation presented above with relation toFIGS. 5-8, the avatars are pictures of their respective faces (although it should be appreciated that visually more intricate avatars may be employed). The user interface display1100of the device presented, by way of example in this figure, shows that the device enables the user to (i) enter and send messages, (ii) receive messages and (ii) dynamically manipulate the selected avatar (thereby effectuating an animation and/or haptic effect). As illustrated and discussed above, when a user's avatar is touched (or interacted with by a user—Bobbi or Roman), the device displays a message indicating the animation/haptic effect triggered, and a message indicating the type of animation/haptic effect1105and1107.

As shown inFIG. 11, Roman1104notices that Bobbi1106has not responded to a text he sent a while ago. He wonders if Bobbi received the text, and wants to notify her in a fun, amusing way, as opposed to sending her repeated messages. Here, for example, Roman1104can press down with his thumb on Bobbi's avatar1106and then shake his device. Immediately, the screen displays an icon that shows an avatar shaking, and next to the icon it says “You shook Bobbi!”1107. Thus, Bobbi's mobile device vibrates, and she checks her phone and sees that Roman “shook” her.

In another non-limiting example in accordance with some embodiments, Bobbi can interact with her own displayed avatar in order to express similar emotions. As with current text features, and understood to be applicable for purposes of this example, when users are having a conversation, each user's avatar is visible to each user and themselves. Instead of having to interact with the other user's avatar, Bobbi can place her finger on her own avatar on the touch screen of her mobile device. This can effectuate the same result as discussed above.

In some embodiments, in addition to manipulating an avatar, a user can also modify an avatar to produce an animation. For example, a user can manipulate or distort the face, or component of the face, of an avatar (or emoticon). For example, a user may insert a smiley face emoticon into a text message and then manipulate or distort the face or a component of the face (e.g. the smile) to broaden the smile into a grin, or twist the smile into an ironic smile. Similarly, the user can modify an avatar in a similar manner. This enables the user to personalize the avatar or emoticon rather than having to select the most appropriate emoticon from a palette of predefined emoticons. Therefore, a user is afforded the ability to modify graphical user representations in real-time to produce personalized moods, emotions, gestures and statements. It should also be understood, that the input and output effectuated via interaction with an avatar is not limited to the avatar as a whole. For example, interaction can be related to solely a component of an avatar, e.g., a person's smile or eyes, or even the background imagery within an avatar's display area.

It should be understood that although discussed above primarily with respect to instant message applications, other implementations are contemplated for providing similar functionality in platforms and online applications. For example, the techniques and concepts may be applied to an animated avatar that acts as an information assistant within games, on social networks, to convey news, weather, and other information to a user of a computer system or a computing device.

FIG. 12is a block diagram illustrating an internal architecture of a computing device, e.g., a computing device such as server or user computing device, in accordance with one or more embodiments of the present disclosure.FIG. 12illustrates a computer system upon which some exemplary embodiments of the present disclosure may be implemented. Although computer system1200is depicted with respect to a particular device or equipment, it is contemplated that other devices or equipment (e.g., network elements, servers, processors) within can deploy the illustrated hardware and components of system1200.

As shown inFIG. 12, internal architecture1200includes one or more processing units, processors, or processing cores, (also referred to herein as CPUs)1212, which interface with at least one computer bus1202. Also interfacing with computer bus1202are computer-readable medium, or media,1206, network interface1214, memory1204, e.g., random access memory (RAM), run-time transient memory, read only memory (ROM), media disk drive interface1220as an interface for a drive that can read and/or write to media including removable media such as floppy, CD-ROM, DVD, media, display interface1210as interface for a monitor or other display device, keyboard interface1216as interface for a keyboard, pointing device interface1218as an interface for a mouse or other pointing device, and miscellaneous other interfaces not shown individually, such as parallel and serial port interfaces and a universal serial bus (USB) interface.

Memory1204interfaces with computer bus1202so as to provide information stored in memory1204to CPU1212during execution of software programs such as an operating system, application programs, device drivers, and software modules that comprise program code, and/or computer executable process steps, incorporating functionality described herein, e.g., one or more of process flows described herein. CPU1212first loads computer executable process steps from storage, e.g., memory1204, computer readable storage medium/media1206, removable media drive, and/or other storage device. CPU1212can then execute the stored process steps in order to execute the loaded computer-executable process steps. Stored data, e.g., data stored by a storage device, can be accessed by CPU1212during the execution of computer-executable process steps.

Persistent storage, e.g., medium/media1206, can be used to store an operating system and one or more application programs. Persistent storage can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, playlists and other files. Persistent storage can further include program modules and data files used to implement one or more embodiments of the present disclosure, e.g., listing selection module(s), targeting information collection module(s), and listing notification module(s), the functionality and use of which in the implementation of the present disclosure are discussed in detail herein.

Network link1228typically provides information communication using transmission media through one or more networks to other devices that use or process the information. For example, network link1228may provide a connection through local network1224to a host computer1226or to equipment operated by a Network or Internet Service Provider (ISP)1230. ISP equipment in turn provides data communication services through the public, worldwide packet-switching communication network of networks now commonly referred to as the Internet1232.

A computer called a server host1234connected to the Internet1232hosts a process that provides a service in response to information received over the Internet1232. For example, server host1234hosts a process that provides information representing video data for presentation at display1210. It is contemplated that the components of system1200can be deployed in various configurations within other computer systems, e.g., host and server.

At least some embodiments of the present disclosure are related to the use of computer system1200for implementing some or all of the techniques described herein. According to one embodiment, those techniques are performed by computer system1200in response to processing unit1212executing one or more sequences of one or more processor instructions contained in memory1204. Such instructions, also called computer instructions, software and program code, may be read into memory1204from another computer-readable medium1206such as storage device or network link. Execution of the sequences of instructions contained in memory1204causes processing unit1212to perform one or more of the method steps described herein. In alternative embodiments, hardware, such as ASIC, may be used in place of or in combination with software. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware and software, unless otherwise explicitly stated herein.

The signals transmitted over network link and other networks through communications interface, carry information to and from computer system1200. Computer system1200can send and receive information, including program code, through the networks, among others, through network link and communications interface. In an example using the Internet, a server host transmits program code for a particular application, requested by a message sent from computer, through Internet, ISP equipment, local network and communications interface. The received code may be executed by processor1202as it is received, or may be stored in memory1204or in storage device or other non-volatile storage for later execution, or both.

For the purposes of this disclosure the term “user”, “subscriber” or “customer” should be understood to refer to a consumer of data supplied by a data provider. By way of example, and not limitation, the term “user” or “subscriber” can refer to a person who receives data provided by the data or service provider over the Internet in a browser session, or can refer to an automated software application which receives the data and stores or processes the data.

As discussed herein, a “username” is an identifier that may be composed of letters, numbers or symbols (collectively defined as alpha-numeric characters) that is used to gain access to, and uniquely identify, an account and/or a network. Generally, but not always, a unique username will be selected by the user that initially registers for use of a message account. Typically, a domain name identifies the provider of the message account for the user, and which is appended to the username and separated by the “@” symbol.