Conversational Assistants For Emergency Responders

A method includes capturing data representing an emergency scene, the data including at least one of image data, sound data, or video data representing the emergency scene. The method also includes generating, using a generative model, based on the data representing the emergency scene, a descriptive summary of the emergency scene. The method further includes providing the descriptive summary of the emergency scene to an emergency responder.

TECHNICAL FIELD

This disclosure relates to conversational assistants for emergency responders.

BACKGROUND

Increasingly, users are using conversational assistants to interact with user devices. A conversational assistant provides a user interface that is configured to mimic interactions with a live person.

SUMMARY

One aspect of the disclosure provides a computer-implemented method for providing a conversational assistant for emergency responders. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations including capturing data representing an emergency scene, the data including at least one of image data, sound data, or video data representing the emergency scene; generating, using a generative model, based on the data representing the emergency scene, a descriptive summary of the emergency scene; and providing the descriptive summary of the emergency scene to an emergency responder.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the generative model includes a classifier model configured to, based on the data representing the emergency scene, identify aspects of the emergency scene, and a natural language processing model configured to, based on the aspects of the emergency scene, generate the descriptive summary of the emergency scene. In some examples, the operations also include providing, using a conversational assistant, the descriptive summary of the emergency scene to the emergency responder.

In some examples, the operations also include detecting an emergency and automatically, in response to detecting the emergency: capturing the data representing the emergency scene, generating the descriptive summary of the emergency, and providing the descriptive summary to the emergency responder. Detecting the emergency may include receiving, from a person, an indication of the emergency. Detecting the emergency may include receiving, from a vehicle involved in the emergency, an indication of the emergency.

In some implementations, the descriptive summary includes at least some of the data representing the emergency scene. The descriptive summary may be generated to assist the emergency responder in preparing for the emergency scene upon arrival. The descriptive summary may include one or more of states of one or more vehicles involved in an accident at the emergency scene; states of one or more airbags; locations of one or more vehicles at the emergency scene; a health status of one or more persons involved in the accident at the emergency scene; locations of one or more persons involved in the accident; a presence of fire; a presence of water; a presence of a roadway; damage to a roadway; a terrain topology; a presence of a leaking fluid; debris on a roadway; a roadblock condition; sounds at the emergency scene including speaking, crying, moaning, or barking; a description of surroundings; or a presence of weapons. In some examples, the descriptive summary includes text and the operations also providing the text to a text-to-speech (TTS) system, the TTS system configured to convert the text into TTS audio data that conveys the descriptive summary as synthetic speech, providing the descriptive summary of the emergency scene to the emergency responder includes providing the TTS audio data to the emergency responder.

In some implementation, capturing the data representing the emergency scene includes capturing data using at least one of a camera or a microphone in communication with a user device associated with a person who is present at the emergency scene. Here, the descriptive summary may include an image of the person to facilitate identification of the person by the emergency responder. Additionally or alternatively, capturing the data representing the emergency scene includes capturing data using at least one of a camera or a microphone of a vehicle involved in the emergency scene. Additionally or alternatively, capturing the data representing the emergency scene includes capturing sensor data from one or more sensors in communication with the data processing hardware, the one or more sensors including at least one of a speed sensor, an altitude sensor, an accelerometer, a braking sensor, a position sensor, a temperature sensor, or a light sensor.

Another aspect of the disclosure provides a system including data processing hardware, and memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, causes the data processing hardware to perform operations. The operations including capturing data representing an emergency scene, the data including at least one of image data, sound data, or video data representing the emergency scene; generating, using a generative model, based on the data representing the emergency scene, a descriptive summary of the emergency scene; and providing the descriptive summary of the emergency scene to an emergency responder.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the generative model includes a classifier model configured to, based on the data representing the emergency scene, identify aspects of the emergency scene, and a natural language processing model configured to, based on the aspects of the emergency scene, generate the descriptive summary of the emergency scene. In some examples, the operations also include providing, using a conversational assistant, the descriptive summary of the emergency scene to the emergency responder.

In some examples, the operations also include detecting an emergency and automatically, in response to detecting the emergency: capturing the data representing the emergency scene, generating the descriptive summary of the emergency, and providing the descriptive summary to the emergency responder. Detecting the emergency may include receiving, from a person, an indication of the emergency. Detecting the emergency may include receiving, from a vehicle involved in the emergency, an indication of the emergency.

In some implementations, the descriptive summary includes at least some of the data representing the emergency scene. The descriptive summary may be generated to assist the emergency responder in preparing for the emergency scene upon arrival. The descriptive summary may include one or more of states of one or more vehicles involved in an accident at the emergency scene; states of one or more airbags; locations of one or more vehicles at the emergency scene; a health status of one or more persons involved in the accident at the emergency scene; locations of one or more persons involved in the accident; a presence of fire; a presence of water; a presence of a roadway; damage to a roadway; a terrain topology; a presence of a leaking fluid; debris on a roadway; a roadblock condition; sounds at the emergency scene including speaking, crying, moaning, or barking; a description of surroundings; or a presence of weapons. In some examples, the descriptive summary includes text and the operations also providing the text to a text-to-speech (TTS) system, the TTS system configured to convert the text into TTS audio data that conveys the descriptive summary as synthetic speech, providing the descriptive summary of the emergency scene to the emergency responder includes providing the TTS audio data to the emergency responder.

In some implementation, capturing the data representing the emergency scene includes capturing data using at least one of a camera or a microphone in communication with a user device associated with a person who is present at the emergency scene. Here, the descriptive summary may include an image of the person to facilitate identification of the person by the emergency responder. Additionally or alternatively, capturing the data representing the emergency scene includes capturing data using at least one of a camera or a microphone of a vehicle involved in the emergency scene. Additionally or alternatively, capturing the data representing the emergency scene includes capturing sensor data from one or more sensors in communication with the data processing hardware, the one or more sensors including at least one of a speed sensor, an altitude sensor, an accelerometer, a braking sensor, a position sensor, a temperature sensor, or a light sensor.

DETAILED DESCRIPTION

Increasingly, users are using conversational assistants to interact with user devices. A conversational assistant provides a user interface that is configured to mimic interactions with a live person. However, conventional conversational assistants do not process or comprehend sounds or visual content of an environment surrounding a user that is interacting with a conversational assistant. To the extent a conventional conversational assistant does process sounds or visual content of an environment surrounding a user, the conversational assistant only does so to isolate utterances of the user from sounds in the environment and/or to identify the user. However, sounds or visual content of an environment may represent information that may be valuable to a conversational interaction with a conversational assistant. For example, an emergency responder interacting with a conversational assistant regarding an emergency scene may benefit greatly from information related to sounds or visual content of an environment that includes the emergency scene.

Implementations herein are directed toward methods and systems capable of providing conversational assistants the ability to process and understand captured sounds or visual content representing a surrounding environment. In particular, implementations herein are directed toward using conversational assistants as an interface between a person present at an emergency scene and an emergency responder that can, based on sounds and/or visual content representing the emergency scene, generate and provide a descriptive summary of the emergency scene to the emergency responder. Notably, the descriptive summary of the emergency scene is generated to provide detailed information that may assist the emergency responder in preparing for the emergency scene upon arrival. For example, such detailed information may expedite the ability provide necessary help more efficaciously, reduce risks of injury or death, save lives, improve public safety, reduce property damage, reduce inconvenience to others, and/or to respond with sufficient personnel and/or equipment.

FIG.1is a schematic view of an example of a system100using a generative model200for generating a descriptive summary202of an emergency scene102. In particular, a conversational assistant120executing on a remote computing system70and/or on a user device10associated with a user101present at the emergency scene102may initiate an emergency communication with an emergency responder104. The conversational assistant120may provide a conversational user interface130for execution on the user device10for facilitating communications related to the emergency communication between the user101and the emergency responder104. The conversational assistant120provides the descriptive summary202of the emergency scene102to the emergency responder104. The generative model200may execute on the user device10or the remote computing system70in communication with the user device10via a network40. In some examples, the generative model200and the conversational assistant120execute together on the user device10or the remote computing system70. Optionally, the conversational assistant120may execute on a device/system different than a device/system where the generative model200executes, but may access the generative model200in the presence of an emergency by providing data112captured at the emergency scene102to the generative model200for processing to generate the descriptive summary202of the emergency scene102. In some implementations, the generative models200includes a deep learning model such as a transformer model, a bidirectional autoregressive transformer (BART) model, or a text-to-text transfer transformer (T5) model. As used herein, the conversational assistant may communicate the descriptive summary202to one or more emergency devices each associated with an emergency responder104. The emergency scene102may include any number and/or type(s) of emergencies, incidents, events, etc. including, but not limited to, health or medical emergencies, accidents, natural disasters, and criminal behaviors. The emergency responder104may be any type of emergency responder including, but not limited to, call center personnel, hotline personnel, police personnel, firefighting personnel, emergency medical technician (EMT) personnel, nurses, and doctors.

The user101may correspond to a live person present at the emergency scene102. For instance, the user101may be an individual involved in the emergency scene and potentially injured as result of the emergency. Here, the user101may manually invoke the conversational assistant120through the user device10to initiate the emergency communication with the emergency responder104by speaking a trigger/command phrase, providing a user input indication indicating selection of a graphical element for initiating emergency communications, or by any other means. In some situations, the conversational assistant120initiates the emergency communication with the emergency responder104without any input from the user101on the user's101behalf. For instance, the user device10may detect a presence of an emergency (e.g., a vehicle crash where sensors of the user device10capture data indicative of the vehicle crash) and invoke the conversational assistant120to initiate the emergency communication with the emergency responder104. The user device101located at the emergency scene102is configured to capture emergency data112(e.g., image data, sound data, or video data) representing the emergency scene102and provide, via the conversational assistant120, the captured data112to the generative model200for processing thereof to generate the descriptive summary202of the emergency scene102. The conversational assistant120may provide the descriptive summary202the emergency responder104. In some implementations, the conversational assistant120(or an initial emergency responder104) performs semantic analysis on the descriptive summary202generated by the generative model200to identify emergency responders104that are most appropriate for responding to the emergency scene102. For instance, the conversational assistant120may provide the descriptive summary202to emergency responders104that include an EMT if the descriptive summary202indicates there are or may be injured people at the emergency scene in need of medical assistance. Likewise, the conversational assistant120may provide the descriptive summary to emergency responders104that include firefighters if the descriptive summary202indicates the presence of a fire at the emergency scene102.

Some examples of user devices10include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., a smart watch, smart glasses, smart goggles, an AR headset, a VR headset, etc.), smart appliances, Internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device10includes data processing hardware12and memory hardware14in communication with the data processing hardware12and stores instructions, that when executed by the data processing hardware12, cause the data processing hardware12to perform one or more operations. The user device10further includes one or more input/output devices16,16a-n,such as an audio capture device16,16a(e.g., microphone) for capturing and converting sounds into electrical signals, an audio output device16,16b(e.g., a speaker) for communicating an audible audio signal (e.g., as output audio data from the user device10), a camera16,16cfor capturing image data (e.g., images or video), and/or a screen16,16dfor displaying visual content. Of course, any number and/or type(s) of other input/output devices16may be used. The input/output devices16may reside on or be in communication with the user device10. For instance, the user device10may execute a graphical user interface17for display on the screen16dthe presents the conversational user interface130provided by the conversational assistant120for facilitating the emergency communication between the user101and the emergency responder. Here, the conversational user interface130may present a textual representation of the descriptive summary202provided to the emergency responder104. Additionally or alternatively, the descriptive summary202and/or the conversational user interface130may include captured visual content112(e.g., one or more images and/or videos) and/or captured audio content112(e.g., one or more audio recordings) of the emergency scene102provided by the user101. The conversational interface130may present a dialog of communications/interactions between the user101and the emergency responder104related to the emergency scene102. The conversational user interface130may permit the user101to interact with the assistant120via speech captured by the microphone16band may optionally present a transcription of the captured speech recognized by an automated speech recognition (ASR) system for display on the screen16d.The conversational assistant120may provide audio data characterizing speech/voice inputs spoken by the user101to the emergency responder104and/or transcriptions of the speech/voice to the emergency responder104. The emergency responder104may be provided a similar conversational user interface130on a device associated with the emergency responder104.

The user device10and/or the remote computing system70(e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device10via the network40executes an input subsystem110configured to receive data112,112a-ncaptured by any number and/or type(s) of data capturing devices (not shown for clarify of illustration) that may reside on any combination of the user device112or other devices in communication with the conversational assistant120. Here, the data112includes at least one of image data, sound data, or video data112representing the emergency scene102that is provided to the generative model200for generating the descriptive summary202of the emergency scene102. Example data capturing devices include, but are not limited to, stationary or mobile cameras, microphones, sensors, traffic cameras, security cameras, satellites, portable user devices, wearable devices, vehicle cameras, and vehicle infotainment systems. The data capturing devices may be owned, operated, or provided by any number and/or type(s) of entities. Example images112of an emergency scene102include, but are not limited to, images of a car involved in an accident, an injured or sick person, a deployed airbag, a shattered window, an injured person or animal, a crying or moaning person, or a barking dog, debris on the ground or road, an unconscious person, fire, flooding, or an active shooter. Example videos112of an emergency scene102include, but are not limited to, videos of a car involved in an accident, an injured or sick person, a deployed airbag, a shattered window, an injured person or animal, a crying or moaning person, or a barking dog, debris on the ground or road, an unconscious person, fire, flooding, or an active shooter. Example audio112of an emergency scene102includes, but is not limited to, sounds of a car involved in an accident, an injured or sick person, a deployed airbag, a shattered window, an injured person or animal, a crying or moaning person, or a barking dog, debris on the ground or road, an unconscious person, fire, flooding, or an active shooter.

In some examples, an emergency at an emergency scene102is automatically detected by processing a stream of monitoring data. Here, detection of the emergency automatically triggers the input subsystem110to store the streaming data112and/or automatically trigger capture of additional or alternative data112. Detection of the emergency triggers the conversational assistant120to control the generative model200to generate a descriptive summary202of the emergency scene102, and to provide the descriptive summary202to the appropriate emergency responder(s)104. The descriptive summary202may include a natural language description that summarizes pertinent details of the emergency scene102based on the captured data112input to the generative model200. Additionally or alternatively, the descriptive summary202may include image data depicting a visual representation of the emergency scene that was captured by the user device10or by another image capture device located at the emergency scene102. As mentioned above, the conversational assistant120may perform semantic interpretation (and/or image analysis on visual data) on the descriptive summary to identify the appropriate emergency responder(s)104for responding to the emergency scene102. In some examples, an emergency responder104includes an emergency contact associated with the user101.

Additionally or alternatively, detecting an emergency may include receiving, at the input subsystem110, an indication of the emergency from a vehicle involved in the emergency. Here, the vehicle may detect the emergency based on sensed data (e.g., activation of an airbag, or a sound of speaking or a scream, moan, cry, bark, glass shattering, debris hitting the ground, or a gunshot,), and a camera or microphone of the vehicle may capture and provide image, sound, or video data112to the input subsystem110. In some examples, the vehicle includes the user device10that captures data112representing a person's responses to queries regarding the emergency, injuries, persons involved, condition, location, etc. Other devices that may similarly detect and indicate emergencies, and provide data112include, but are not limited to, barriers, bridges, motion sensors, water level sensors, and impact sensors.

Additionally or alternatively, detecting an emergency may include receiving an indication of the emergency from a person present at the emergency scene102(e.g., the user101involved in or a bystander to an emergency). Here, the user101may trigger the user device10to capture of image, sound, or video data112using, for example, the camera16cor the microphone16bin communication with the user device10associated with the user101. The user101may use the user device10to report or indicate the emergency and provide the captured data112to the input subsystem110. In some examples, data112captured by the user device10includes a picture of the user101that is included in the descriptive summary202to facilitate identification of the person by an emergency responder104.

Additionally or alternatively, detecting an emergency may include receiving an indication of the emergency from a safety check system, such as those used to monitor ill or elderly persons. Here, if a person fails to respond to a safety check, the safety check system may trigger capture of image, sound, or video data112using, for example, a camera or a microphone in communication with the user device10associated with the user101that includes the ill or elderly person being monitored, report or indicate the emergency, and provide the captured data112to the input subsystem110.

The generative model200is configured to processes the data112representing the emergency scene102to generate the descriptive summary202of the emergency scene106. The descriptive summary202of the emergency scene102may be provided as text202,202t(seeFIG.2) and/or as TTS audio data202,202a(seeFIG.2) that conveys the descriptive summary202as synthetic speech. The descriptive summary202may also include some or all of the data112. The data112may include at least one of image data, sound data, or video data112representing the emergency scene102. Additionally or alternatively, the data112may include sensor data from one or more sensors including, but not limited to, a speed sensor, an altitude sensor, an accelerometer, a braking sensor, a position sensor, a temperature sensor, or a light sensor.

A descriptive summary202of an emergency scene102may include, for example, states of one or more vehicles involved in an accident at the emergency scene; states of one or more airbags; locations of the one or more vehicles at the emergency scene; a health status of one or more persons or animals involved in the accident at the emergency scene; locations of the one or more persons or animals involved in the accident; a presence of fire; a presence of water; a presence of a roadway; damage to a roadway; a terrain topology; a presence of a leaking fluid; debris on a roadway; a roadblock condition; sounds at the emergency scene including speaking, crying, moaning, or barking; a description of surroundings; or a presence of weapons, event timestamps, a type of emergency, a number of people or animals involved, or type of assistance required. The descriptive summary202may include a natural language summary of the emergency scene to convey details of the emergency scene102that are pertinent for assisting emergency responders104for responding to the emergency scene102upon arrival.

FIG.3Adepicts an example descriptive summary202A generated by the generative model200for a car accident. A computing device associated with the emergency responder104may display the descriptive summary202A and/or audibly output the descriptive summary202A as synthesized speech.FIG.3Bdepicts an example descriptive summary202B generated by the generative model200for a gunshot incident. A computing device associated with the emergency responder104may display the descriptive summary202B and/or audibly output the descriptive summary202B as synthesized speech.

Returning toFIG.1, the conversational assistant120is configured to provide a conversational user interface130that includes conversational, human-like interactions between the emergency responder104and the conversational assistant120and/or the generative model200. Here, the conversational assistant120may include any number and/or type(s) of models, methods, algorithms, systems, software, hardware, instructions, etc. configured to mimic human-like conversations and/or interactions with the emergency responder104. The emergency responder's manner of interacting with the user device10may be through any number and/or type(s) of inputs. Example inputs include, but are not limited to, text (e.g., entered using a physical or virtual keyboard), spoken utterances, video (e.g., representing gestures or expressions), and clicks (e.g., using a mouse or touch inputs). The conversational assistant120is configured to capture and respond to inputs from the emergency responder104. Notably, the conversational assistant120is able to naturally interact with emergency responders104, and is able to understand and respond to spoken or text-based commands and queries. The conversational assistant120may also be able to learn and adapt over time as it interacts with more emergency responders104and processes data112for more emergencies to improve identifying and responding to emergencies.

The user device10and/or the remote computing system70also executes a user interface generator140configured to provide, for output on an output device of the user device10(e.g., on the audio output device(s)16aor the display16d), entries/responses122of the user101, conversational assistant120, and/or the emergency responder104. Here, the user interface generator140displays the entries and the corresponding responses122in the conversational user interface130. In the illustrated example, the conversational user interface130is for an interaction between the user101, or the conversational assistant120on the user's behalf, and an emergency responder104. However, the conversational user interface130may also be for a multi-party chat session including interactions amongst multiple emergency responders104and the conversational assistant120.

FIG.2is a schematic view of an example of a generative model200. The generative model200executes a classifier model210configured to process data112representing an emergency scene102to identify aspects212of the emergency scene102, and a natural language processing (NLP) model220configured to, based on the aspects212of the emergency scene102identified by the classifier model210, generate the descriptive summary202of the emergency scene102. The NLP model220may include a large language model (LLM). Here, the NLP model220may, based on the aspects212of the emergency scene102, determine a course of action or response to an emergency. For example, when the classifier model210detects aspects212of life threatening activities, the NLP model220could indicate that the conversational assistant120should contact a police office or medical responder. In another example, when the classifier model210detects aspects212of a vehicle crash, the NPL model220could provide information for emergency personnel, such as a location of the incident, a number of people involved, and a summary of the emergency scene102. As described below in greater detail below with reference toFIG.5, the classifier model210and the NLP model220may be trained on a supervised and/or an unsupervised training data set that includes sounds, images, videos, and/or descriptions of emergencies.

In some examples, the descriptive summary202includes text202tprovided by the conversational assistant120to the emergency responder104. Additionally or alternatively, the generative model200includes a text-to-speech (TTS) system230configured to convert the text202tinto TTS audio data202athat conveys the descriptive summary202as synthetic speech, and the conversational assistant120provides the TTS audio data202ato the emergency responder104. Here, the TTS audio data202amay include spectrograms, and/or a time sequence of audio waveform data representing the synthetic speech. The descriptive summary202may also include the data112including at least one of image data, sound data, or video data representing the emergency scene102.

FIG.4is a flowchart of an exemplary arrangement of operations for a computer-implemented method400of providing a descriptive summary202of an emergency scene102. The operations may be performed by data processing hardware710(FIG.7) (e.g., the data processing hardware12of the user device10or the data processing hardware72of the remote computing system70) based on executing instructions stored on memory hardware720(e.g., the memory hardware14of the user device10or the memory hardware74of the remote computing system70).

At operation402, the method400includes capturing data112representing an emergency scene102. The data112may include image data, sound data, or video data representing the emergency scene102. At operation404, the method400includes generating, using the generative model200, based on the data112representing the emergency scene102, a descriptive summary202of the emergency scene102. At operation406, the method400includes the conversational assistant120providing the descriptive summary202of the emergency scene102to an emergency responder104.

FIG.5is a flowchart of an exemplary arrangement of operations for another computer-implemented method500of providing a descriptive summary202of an emergency scene102. The operations may be performed by data processing hardware710(FIG.7) (e.g., the data processing hardware12of the user device10or the data processing hardware72of the remote computing system70) based on executing instructions stored on memory hardware720(e.g., the memory hardware14of the user device10or the memory hardware74of the remote computing system70).

At operation502, the method500includes the conversational assistant120receiving captured data112representing a potential emergency scene102. The data112may include image data (e.g., of a car crash, an injured or sick person or animal, an unconscious person, fire, a firearm, a flood, etc.), sound data (e.g., of an airbag inflating, a child crying, moaning in pain, a dog barking, a person asking for assistance, etc.), or video data (e.g., of an injured or sick person or animal, an unconscious person, fire, a firearm, a flood, etc.) representing the emergency scene102.

At operation504, the method500includes the classifier model210processing the received data112to automatically analyze the scene102for an emergency. Additionally or alternatively, the conversational assistant detects, operation504, an emergency at the scene102based on the user101reporting the emergency. When a decision is affirmative (“YES”) that the emergency is detected at operation506, the method500includes, at operation508, the generative model200(i.e., the NLP/LLM model220) processing the data112to generate the descriptive summary202of the emergency scene102. At operation510, the method500includes the conversational assistant120initiating a911call or sending a message to a911service for providing the descriptive summary202of the emergency scene102to an emergency responder104. Additionally or alternatively, the method500may include, at operation512, the conversational assistant120sending a cloud link to a911service for retrieving the descriptive summary202. Here, the generative model200may, based on additional or updated data112representing the emergency scene102, generate an updated descriptive summary202of the emergency scene102.

FIG.6is a flowchart of an exemplary arrangement of operations for a computer-implemented method600of training the generative model200for generating a descriptive summary202of an emergency scene102. The operations may be performed by data processing hardware710(FIG.7) (e.g., the data processing hardware12of the user device10or the data processing hardware72of the remote computing system70) based on executing instructions stored on memory hardware720(e.g., the memory hardware14of the user device10or the memory hardware74of the remote computing system70).

At operation602, the method600includes obtaining a data set representing a plurality of emergency scenes102. Here, the data set includes, for each particular emergency scene102, corresponding captured sound, image, and/or video data112representing the particular emergency scene102. Here, the data set may represent a diverse and large set of emergency scenes102to ensure the generative model200can generalize well to new emergency scenes102.

At operation604, the method600includes, for each particular emergency scene102, labeling the particular emergency scene102with a corresponding ground-truth emergency scene type and a corresponding ground-truth descriptive summary202.

At operation606, the method600includes training the generative model200using supervised learning on a first training portion of the data set. Here, training the generative model200includes inputting the data112for each particular emergency scene102into the generative model200and adjusting coefficients on the generative model200until it can accurately predict the corresponding ground-truth emergency scene type and the corresponding ground-truth descriptive summary202. Notably, the classifier model210may be trained to predict the corresponding ground-truth emergency scene type and the NLP/LLM model220may be trained to predict the corresponding ground-truth descriptive summary202. In some examples, the generative model200includes a deep learning model trained on the data set to predict both an emergency scene type and a descriptive summary from input data112for a particular emergency scene202. IN these examples, the generative model200may include, without limitation, a transformer model, a bidirectional autoregressive transformer (BART) model, or a text-to-text transfer transformer (T5) model. At operation608, the method600may optionally test the generative model on a second test portion of the data set that was withheld from the first training portion (i.e., unseen data).

The computing device700includes a processor710(i.e., data processing hardware) that can be used to implement the data processing hardware12and/or72, memory720(i.e., memory hardware) that can be used to implement the memory hardware14and/or74, a storage device730(i.e., memory hardware) that can be used to implement the memory hardware14and/or74, a high-speed interface/controller740connecting to the memory720and high-speed expansion ports750, and a low speed interface/controller760connecting to a low speed bus770and a storage device730. Each of the components710,720,730,740,750, and760, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor710can process instructions for execution within the computing device700, including instructions stored in the memory720or on the storage device730to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display780coupled to high speed interface740. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices700may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device730is capable of providing mass storage for the

computing device700. In some implementations, the storage device730is a computer-readable medium. In various different implementations, the storage device730may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-or machine-readable medium, such as the memory720, the storage device730, or memory on processor710.

The high speed controller740manages bandwidth-intensive operations for the computing device700, while the low speed controller760manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller740is coupled to the memory720, the display780(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports750, which may accept various expansion cards (not shown). In some implementations, the low-speed controller760is coupled to the storage device730and a low-speed expansion port790. The low-speed expansion port790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device700may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server700aor multiple times in a group of such servers700a,as a laptop computer700b,or as part of a rack server system700c.

The processes and logic flows described in this specification can be performed

Unless expressly stated to the contrary, the phrase “at least one of A, B, or C” is intended to refer to any combination or subset of A, B, C such as: (1) at least one A alone; (2) at least one B alone; (3) at least one C alone; (4) at least one A with at least one B; (5) at least one A with at least one C; (6) at least one B with at least C; and (7) at least one A with at least one B and at least one C. Moreover, unless expressly stated to the contrary, the phrase “at least one of A, B, and C” is intended to refer to any combination or subset of A, B, C such as: (1) at least one A alone; (2) at least one B alone; (3) at least one C alone; (4) at least one A with at least one B; (5) at least one A with at least one C; (6) at least one B with at least one C; and (7) at least one A with at least one B and at least one C. Furthermore, unless expressly stated to the contrary, “A or B” is intended to refer to any combination of A and B, such as: (1) A alone; (2) B alone; and (3) A and B.