Patent ID: 12190015

DETAILED DESCRIPTION

Note that this disclosure relates generally to computer ecosystems including aspects of consumer electronics (CE) device networks such as but not limited to home entertainment networks. A system herein may include server and client components which may be connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including game consoles such as Sony PlayStation® or a game console made by Microsoft or Nintendo or other manufacturer, extended reality (XR) headsets such as virtual reality (VR) headsets, augmented reality (AR) headsets, display devices such as televisions such as the Sony Bravia TVs (e.g., smart TVs, Internet-enabled TVs), computers such as laptops and tablet computers, standalone speakers, home theater audio/video (AV) devices, and mobile devices including the Sony Xperia mobile phone, smart phones and additional examples discussed below. These client devices may operate with a variety of operating environments. For example, some of the client computers may employ, as examples, Linux operating systems, operating systems from Microsoft, or operating systems from Apple, Inc. or Google. These operating environments and in addition with software drivers and a hardware abstraction layer (HAL) may be used to execute one or more browsing programs, such as a browser made by Microsoft or Google or Mozilla or other Internet networked applications that can access websites hosted by the Internet servers discussed below. Also, an operating environment according to present principles may be used to execute one or more computer game programs.

Servers and/or gateways may be used that may include one or more processors executing instructions that configure the servers to receive and transmit data over a network such as the Internet. Or a client and server can be connected over a local intranet or a virtual private network. A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.

Information may be exchanged over a network between the clients and servers. To this end and for security, servers and/or clients can include firewalls, load balancers, temporary storages, and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implement methods of providing a secure community such as an online social website or gamer network to network members.

A processor may be a single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.

Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged, or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together.

Referring now toFIG.1, an example system10is shown, which may include one or more of the example devices mentioned above and described further below in accordance with present principles. The first of the example devices included in the system10is a consumer electronics (CE) device such as an audio video device (AVD)12such as but not limited to a theater display system which may be projector-based, or an Internet-enabled TV with a TV tuner (equivalently, set top box or dongle controlling a TV). The AVD12alternatively may also be a computerized Internet enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and/or headset such as smart glasses or a VR headset, another wearable computerized device, a computerized Internet-enabled music player, computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as an implantable skin device, etc. Regardless, it is to be understood that the AVD12is configured to undertake present principles (e.g., communicate with other CE devices to undertake present principles, execute the logic described herein, and perform any other functions and/or operations described herein).

Accordingly, to undertake such principles the AVD12can be established by some, or all of the components shown. For example, the AVD12can include one or more touch-enabled displays14that may be implemented by a high definition or ultra-high definition “4K” or higher flat screen. The touch-enabled display(s)14may include, for example, a capacitive or resistive touch sensing layer with a grid of electrodes for touch sensing consistent with present principles.

The AVD12may also include an analog audio output port15to drive one or more external speakers or headphones, and may include one or more internal speakers16for outputting audio in accordance with present principles, and at least one additional input device18such as an audio receiver/microphone for entering audible commands to the AVD12to control the AVD12. The example AVD12may also include one or more network interfaces20for communication over at least one network22such as the Internet, an WAN, an LAN, etc. under control of one or more processors24. Thus, the interface20may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as but not limited to a mesh network transceiver. It is to be understood that the processor24controls the AVD12to undertake present principles, including the other elements of the AVD12described herein such as controlling the display14to present images thereon and receiving input therefrom. Furthermore, note the network interface20may be a wired or wireless modem or router, or other appropriate interface such as a wireless telephony transceiver, or Wi-Fi transceiver as mentioned above, etc.

In addition to the foregoing, the AVD12may also include one or more input and/or output ports26such as a high-definition multimedia interface (HDMI) port or a universal serial bus (USB) port to physically connect to another CE device and/or a headphone port to connect headphones to the AVD12for presentation of audio from the AVD12to a user through the headphones. For example, the input port26may be connected via wire or wirelessly to a cable or satellite source26aof audio video content. Thus, the source26amay be a separate or integrated set top box, or a satellite receiver. Or the source26amay be a game console or disk player containing content. The source26awhen implemented as a game console may include some or all of the components described below in relation to the CE device48.

The AVD12may further include one or more computer memories/computer-readable storage media28such as disk-based or solid-state storage that are not transitory signals, in some cases embodied in the chassis of the AVD as standalone devices or as a personal video recording device (PVR) or video disk player either internal or external to the chassis of the AVD for playing back AV programs or as removable memory media or the below-described server. Also, in some embodiments, the AVD12can include a position or location receiver such as but not limited to a cellphone receiver, GPS receiver and/or altimeter30that is configured to receive geographic position information from a satellite or cellphone base station and provide the information to the processor24and/or determine an altitude at which the AVD12is disposed in conjunction with the processor24.

Continuing the description of the AVD12, in some embodiments the AVD12may include one or more cameras32that may be a thermal imaging camera, a digital camera such as a webcam, an IR sensor, an event-based sensor, and/or a camera integrated into the AVD12and controllable by the processor24to gather pictures/images and/or video in accordance with present principles. Also included on the AVD12may be a Bluetooth® transceiver34and other Near Field Communication (NFC) element36for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

Further still, the AVD12may include one or more auxiliary sensors38that provide input to the processor24. For example, one or more of the auxiliary sensors38may include one or more pressure sensors forming a layer of the touch-enabled display14itself and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Other sensor examples include a pressure sensor, a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, an event-based sensor, a gesture sensor (e.g., for sensing gesture command). The sensor38thus may be implemented by one or more motion sensors, such as individual accelerometers, gyroscopes, and magnetometers and/or an inertial measurement unit (IMU) that typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the location and orientation of the AVD12in three dimension or by an event-based sensors such as event detection sensors (EDS). An EDS consistent with the present disclosure provides an output that indicates a change in light intensity sensed by at least one pixel of a light sensing array. For example, if the light sensed by a pixel is decreasing, the output of the EDS may be −1; if it is increasing, the output of the EDS may be a +1. No change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

The AVD12may also include an over-the-air TV broadcast port40for receiving OTA TV broadcasts providing input to the processor24. In addition to the foregoing, it is noted that the AVD12may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver42such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD12, as may be a kinetic energy harvester that may turn kinetic energy into power to charge the battery and/or power the AVD12. A graphics processing unit (GPU)44and field programmable gated array46also may be included. One or more haptics/vibration generators47may be provided for generating tactile signals that can be sensed by a person holding or in contact with the device. The haptics generators47may thus vibrate all or part of the AVD12using an electric motor connected to an off-center and/or off-balanced weight via the motor's rotatable shaft so that the shaft may rotate under control of the motor (which in turn may be controlled by a processor such as the processor24) to create vibration of various frequencies and/or amplitudes as well as force simulations in various directions.

A light source such as a projector such as an infrared (IR) projector also may be included.

In addition to the AVD12, the system10may include one or more other CE device types. In one example, a first CE device48may be a smart speaker assembly or other device and may not have a video display. Or, the first CE device48may include a computer game console that can be used to send computer game audio and video to the AVD12via commands sent directly to the AVD12and/or through the below-described server. A second CE device50may include similar components as the first CE device48. In the example shown, the second CE device50may be configured as a remote control (RC) for the AVDD12. Or, the second CE device50may be configured as a computer game controller manipulated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a heads-up transparent or non-transparent display for respectively presenting AR/MR content or VR content (more generally, extended reality (XR) content). The HMD may be configured as a glasses-type display or as a bulkier VR-type display vended by computer game equipment manufacturers.

In the example shown, only two CE devices are shown, it being understood that fewer or more devices may be used. A device herein may implement some or all of the components shown for the AVD12. Any of the components shown in the following figures may incorporate some or all of the components shown in the case of the AVD12.

Now in reference to the afore-mentioned at least one server52, it includes at least one server processor54, at least one tangible computer readable storage medium56such as disk-based or solid-state storage, and at least one network interface58that, under control of the server processor54, allows for communication with the other illustrated devices over the network22, and indeed may facilitate communication between servers and client devices in accordance with present principles. Note that the network interface58may be, e.g., a wired or wireless modem or router, Wi-Fi or Ethernet transceiver, or other appropriate interface such as, e.g., a wireless telephony transceiver.

Accordingly, in some embodiments the server52may be an Internet server or an entire server “farm” and may include and perform “cloud” functions such that the devices of the system10may access a “cloud” environment via the server52in example embodiments for, e.g., network gaming applications. Or the server52may be implemented by one or more game consoles or other computers in the same room as the other devices shown or nearby.

The components shown in the following figures may include some or all components shown in herein. Any user interfaces (UI) described herein may be consolidated and/or expanded, and UI elements may be mixed and matched between UIs.

Now in reference toFIG.2, suppose an end-user200is hiking along a hiking trail202while listening to an indexed audio book or other type of audio file (e.g., indexed podcast or other recording of one or more people speaking). In the present example, the audio book is a fictional sci-fi novel called “Life on Mars”, which is directed to future human colonization of Mars. The audio book may be stored locally on a wearable device204borne by the user200, which in the present example is a smart watch. Additionally or alternatively, the book may be accessed by the device204over the Internet if, for example, the audio book is stored at a cloud-based Internet server. Once accessed, the device204may use Bluetooth or other wireless communication technology to stream audio of the book (including audible narration and quotes from the book's characters, for example) to ear bud headphones206in the ears of the user200so that the user200can hear the audio book.

Also suppose perFIG.2that the end-user200provides audible input as detected by a microphone on one or more of the devices204,206, and that the devices204and/or206process the audible input using a voice assistant application (“app”) executing at one or both devices. In various examples, the voice assistant app may be a digital assistant configured with voice processing/natural language understanding functionality and, as such, may be similar to Amazon's Alexa, Google's Assistant, or Apple's Siri. Although not shown inFIG.2, note that the assistant may be woken up with a keyword, such as “Audible”, “Rakuten” or “Libby”. However, since it may be assumed that the audiobook application is playing, the digital assistant may already be awake and can respond to commands. It is envisioned that the digital assistant would be able to respond to queries regarding plot from the beginning, from the last action scene, or just the current scene. And these can be in the form of summaries that might be organized chapter by chapter or simply a linear recitation of events. In addition, a user may ask questions about any object, technology, characters as well as any groups, organizations and locations cited. The digital assistant may be able to detect that the user is talking and automatically pause playback. Alternatively, an explicit command can be given. The command can be given using natural language. This can all be done without being a spoiler. For example, if a character is shown to be a spy in a subsequent chapter of an espionage thriller, the description up the current playback point would omit that detail. One or both of the devices204,206may then process the audible input to identify a pause command (e.g., “Can you pause” as shown via the speech bubble208) and to identify ensuing voice input requesting a summary and/or other information (e.g., “Summarize chapters one and two. And who is Levi?”). Responsive to receipt of the pause command, the device(s) may pause audible presentation of the audio book at its current playback position and subsequently provide an audible response through the speakers in the headphones206. The audible response may be dynamically determined by the devices204/206and/or a cloud server in communication with those devices as set forth herein. Thus, in the present example speech bubble210indicates that the audible response is “In chapter one, the earthlings land on Mars using a nuclear-powered spaceship that took only 45 days from earth. In chapter two, the earthlings begin to establish a colony presence on Mars. Levi was the person who led NASA before the current director, James.”

Responsive to providing the audible response, the device may autonomously begin presenting the audio of the audio book again from the current playback position at which the book was paused (e.g. without receiving an unpause or play command from the user200). Or in other examples, responsive to providing the audible response the device may wait a threshold amount of time such as ten seconds to see if the user200asks additional questions/requests, and then responsive to expiration of the threshold amount of time without additional questions/requests being received the device may resume playback of the audio book from the current playback position. Or as yet another example by itself or as combined with the ones above, the device may auto-resume playback after the audible response responsive to receipt of an audible command or other type of command to resume playback (e.g., an audible “unpause” or “play” command).

FIG.3shows additional example aspects consistent with present principles. Specifically,FIG.3shows an example graphical user interface (GUI)300that may be presented on a display of a device operating consistent with present principles. For example, the GUI300may be presented on the display of the wearable device204while the user200hikes along the hiking path202.

As shown inFIG.3, the GUI300may include an indicator302of the title of the audio book being played out, which again is “Life on Mars”. The GUI300may also include a prompt304, which in this example indicates “Got a question? Want a dynamic plot/scene summary? Pause and ask audibly or enter here:”, with “enter here” referencing a text entry field/box306at which the user may enter text using a hard or soft keyboard or vice input to ask a question or otherwise request information. Thus, the user200may pause presentation of the audio book by providing an audible “pause” command as described above and/or by selecting the pause selector308presented as part of the GUI300(or not shown, in some implementation by simply speaking). This in turn may cause playback to stop at its current position, as indicated by the playback timeline310for the current chapter that is being played out (Chapter Three here). In the present example, the current playback position is four minutes, thirty five seconds into the track for Chapter Three, as indicated by the slider312that progresses along the playback time310as the audio of Chapter Three is played out in real time to indicate the current playback position. Thus, after providing a pause command one way or another, the user may enter text into text input box (or speak the request audibly) and then select the submit selector314to submit the question/request to the device for processing.

The GUI400ofFIG.4shows an example output that may then be provided by the device in response. As shown, the GUI400may first present an indication402of the question itself as provided by the user, which in the present example is “What has happened in the current chapter up until this point?” As also shown inFIG.4, the GUI400may include dynamically-generated text404indicating an answer to the user's question, with it being further understood that the dynamically-generated text404may also be read aloud to the user via one or more speakers (e.g., speakers on the headphones206). For example, the text404may be read aloud using a text-to-speech algorithm to present the text audibly using a computer-generated voice. It should be understood that the creators of the audio book may wish to use the same narrator for the responses as for the narration of the book itself. In which case audio clips would be recorded for each response, and the audio clips would be played out. Alternatively, the computer-generated voice could be made to sound like the human narrator.

As for the text404itself, in the present example the text indicates “Answer: The earthlings start building indoor farms on Mars (0:00-1:15). Then they start building factories and bringing more workers from Earth (1:15-4:30). James and Levi speak the most during these events.” As may be appreciated from the foregoing, different aspects of this plot summary for Chapter Three may be indexed with metadata for different playout time ranges respectively corresponding to each of those aspects, as may be already configured and stored in storage accessible to the client device of the user (e.g., the wearable device204).

So, for example, a metadata engineer, system administrator, or audio book publisher might associate, in metadata for the audio book, different plot summary elements of an overall plot summary for the audio book with different playback position ranges for what portions of the audio itself are associated with those respective elements of the plot summary. Then during execution of whatever media player app is being used to present the audio book and based on the user's request, the device/media player app may determine the current playback position as tracked by the device/app itself and then execute natural language processing (e.g., topic segmentation and natural language understanding) to determine the specific type of information being requested. The device/app may then lookup the requested information by accessing and parsing the prestored metadata to dynamically generate a conforming summary that summarizes one or more events up to the current playback position but not events associated with playback positions subsequent to the current playback position. These technical procedures may therefore provide a cognitive aid to the user without spoiling or otherwise revealing other content of the book from latter portions that are juxtaposed after the current playback position. And note that although plot summary has been discussed above, biographies for the characters/individuals in the book as well as glossary terms/descriptions for inanimate objects in the book may also have different portions thereof tagged with metadata of respective playback position ranges to which the respective portions of the biographies/glossary terms pertain so that the device may also autonomously and dynamically provide conforming outputs when questions or requests from the user invoke those elements of the audio book.

Still in reference toFIG.4, note further that certain words from the dynamically-generated summary404may be listed on the face of respective selectors406,408. The selectors406,408may be selectable using voice input, touch input, cursor input, etc. to, in response to selection, provide additional outputs regarding the subject of the associated text itself. In the present example and using the same techniques described above, the selector406for the book's character James may be selected to output bibliographic and event data regarding James so that only bibliographic and event data about James that has been associated with playback time ranges up until the current playback position may be output. This information may be output audibly and/or output visually through another GUI like the GUI400. The bibliographic data may include background and other information about James as stated in the audio book itself, and the event data may include events from the audio book up until the current playback position in which James has participated.

As also shown inFIG.4, selector408may be selected to command the device to output similar bibliographic and event data about a different character from the audio book (Levi) that is mentioned in the dynamically-generated summary404. Further note that an inanimate object in a dynamically-generated summary that is output to the user may also be referenced via an associated selector for that object. In response to selection of such a selector, the device may then similarly provide an object description and event data about that inanimate object. For example, the inanimate object may be a spaceship named “Marslink 2”, and selection of a “Marslink 2” selector from a summary like the summary404may command the device to present an output describing Marslink 2 as a NASA spaceship made on earth that traveled to Mars as part of the plot of the audio book. And further note here that these types of character and inanimate object outputs may be provided not just based on selection of a selector from a GUI but also based on a separate audible request for such outputs consistent with present principles.

Further note according toFIG.4that when the user is ready to resume playback of the audio book from the current playback position, as might occur in embodiments where playback was paused while the summary404was generated and output, the user may select the play/return selector410to command the device to resume playback.

Now in reference toFIG.5, example logic is shown that may be executed by a device such as the system10, wearable device204, server52, etc. in any appropriate combination consistent with present principles. Note that while the logic ofFIG.5is shown in flow chart format, other suitable logic may also be used.

Beginning at block500, the device may play out an audio book (or other type of audio file) through one or more speakers. At block500the device may also monitor/track the current playback position of the audio book. The logic may then proceed to decision diamond501where the device may check the status of the604flag to determine whether a pause flag has been set (e.g., “Don't Respond unless pause command is received first”). If a negative determination is made at diamond501, the logic may proceed to decision diamond504as will be described shortly. However, responsive to an affirmative determination at diamond501, the logic may first proceed to decision diamond502.

At decision diamond502the device may in some examples determine whether a pause command has in fact been received. In these examples, a negative determination at diamond502may cause the logic to revert back to block500and proceed therefrom. However, responsive to an affirmative determination at diamond502(e.g., an audible command to pause audible presentation of the audio book has been identified based on input from a microphone), the logic may proceed to diamond504.

Before describing diamond504in detail, note here that in certain non-limiting example implementations a pause command may be required before processing an ensuing audible request in order to reduce false positives that might otherwise occur should the user, absent a pause command, ask a rhetorical question like “What is happening?” or “How did we get here?” So in these examples, based on identification of the pause command, the device may process a subsequent audible request that is received within a threshold amount of time of the pause request itself (e.g., within ten seconds, otherwise a timeout may occur where audible requests are not processed after the threshold amount of time to further reduce false positives). But further note that based on not identifying a pause command in this example implementation, the device may instead decline to process an ensuing audible request (e.g., at least until an audible command to pause audible presentation of the audio book is received).

Now in reference to diamond504, at this step the device may determine whether there exists and/or actually receive audible input or other input (e.g., text-based input). A negative determination may result in the logic returning to block500, while an affirmative determination may instead cause the logic to proceed to block509to pause the audio book if not already paused and then move on to block506for other action.

For example, input from a microphone may be received at diamond504to identify, at block506based on the input from the microphone, an audible request for a plot summary pertaining to some but not all of the audio book. For example, at block506the device may execute a voice assistant and/or other software to identify the request for the plot summary. Note here that the request might additionally or alternatively request output regarding other aspects of the audio book as well, such as bibliographic data regarding a particular character from the audio book and/or a description of an inanimate object from the audio book.

From block506the logic may then proceed to block508. At block508the device may access metadata for the audio book and execute natural language understanding (NLU) (and/or other natural language processing (NLP) techniques) to identify the requested output from the metadata using NLU so that the requested output is tailored to the content of the user's request using data tagged as occurring within the audio book at or before the current playback position as described herein. From block508the logic may then proceed to block510. At block510the device may audibly and/or visually present the requested plot summary output and/or other requested output in conformance with the request itself. Again note that the request may have been an audible request in non-limiting examples, that the output may summarize the book's plots/events/scenes up to the current playback position, and that the output may also relate to individuals and/or inanimate objects from the audio book.

Continuing the detailed description in reference toFIG.6, it shows an example settings GUI600that may be presented on a display (and/or have its contents audibly read aloud) for an end-user to configure one or more settings of a device to operate consistent with present principles. For example, the GUI600may be presented by navigating a device menu or media player app menu at a client device that is being or will be used for audio book playback. In the example shown, each setting may be selected by directing touch, cursor, or other input to the associated check box adjacent to the respective option.

As shown inFIG.6, the GUI600may include a setting602that may be selectable a single time to set or enable the device to, in the future, undertake present principles for multiple future instances/playbacks of one or more audio books. For example, selection of the setting602may configure the device to execute the logic ofFIG.5, execute the functions described in reference toFIG.2, and to present the GUIs300and400ofFIGS.3and4.

As also shown inFIG.6, the GUI600may include a second setting604that may be selectable to specifically set or enable the device to only respond to audible requests for plot summaries and other audio book outputs if a pause command is received first as described above.

Before concluding, it is to be understood that outputs related to an audio book may include more than just the examples above. For example, in addition to plot/story line summaries, other outputs derived from audio book metadata may be presented upon request including outputs identifying a person that is currently speaking as part of the audio book (e.g., the name of a narrator or simply an output indicating that the narrator is speaking, the name of a particular character from the audio book that is currently speaking, etc.).

It may now be appreciated that present principles provide for an improved computer-based user interface that increases the functionality and ease of use of the devices disclosed herein. The disclosed concepts are rooted in computer technology for computers to carry out their functions.

While the particular embodiments are herein shown and described in detail, it is to be understood that the subject matter which is encompassed by the present invention is limited only by the claims.