Patent ID: 12219029

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference is presently made in detail to exemplary embodiments of the present subject matter, one or more examples of which are illustrated in or represented by the drawings. Each example is provided by way of explanation of the present subject matter, not limitation of the present subject matter. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made in the present subject matter without departing from the scope or spirit of the present subject matter. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present subject matter covers such modifications and variations as come within the scope of the disclosure and equivalents thereof.

FIG.1shows an exemplary embodiment of a system10in accordance with the present invention. Server20receives data from a plurality of user devices such as image and video data from device30, voice and text data from device40, temperature and other data from device50, and location, altitude, and speed data from device60. Thus device30includes a sensor such as a microphone and a camera, device40includes a sensor such as a microphone and a keyboard for receiving text data that may be embodied by a touchscreen displaying the keyboard, device50includes a sensor such as a thermometer, and device60includes a device such as a global positioning system (GPS) sensor. Devices30,40,50, and60may be smartphones, tablets, digital cameras, laptop or desktop computers or any other electronic device capable of collecting and transmitting this data. Further, any of the user devices30,40,50, and60may include more than one sensor, or all of the sensors listed above. In general, each of devices30,40,50, and60will include at least one sensor, a processor, memory, a transmitter for transmitting the data to server20, and a receiver for receiving data from server20. Server20also includes a processor, a memory, a transmitter for transmitting data to devices30,40,50, and60, and a receiver for receiving data from devices30,40,50, and60. Devices30,40,50, and60will be referred to hereinafter as end point devices (EPDs).

In one embodiment, the EPDs are portable electronic devices that run one of the Android®, iOS®, or Blackberry® operating systems. An app run by the device performs the functions described herein as performed by the EPD. An exemplary portable electronic device is a wearable electronic device including a video camera and microphone described in U.S. patent application Ser. No. 13/212,686. This application is incorporated by reference herein. IN another embodiment, the EPD may be a wearable (body worn) health tracking device such as the FitBit®, Pebble®, Basis Peak®, etc.

Server20may include a complex event modeler70and a predictive modeling tool80which analyzes the data received from the devices to determine if the data received from the devices corresponds to an event such as an emergency. The event can be a crime in progress, a severe weather event, or any emergency scenario where life or human/property security (e.g., child abduction, car break-in, arson, tornado, flash mob, etc.) is already or about to be imperiled. Server20manages and transforms event files and automatically generates notifications, including machine to machine (M2M) notifications, using a computer aided dispatch (CAD) tool90, analytic tools, or command and control tools; and/or machine to person (M2P) notifications to a private or public actor100to respond to the event by sending an alert(s) to the actor100with information about the event derived from the uploaded data. The private or public actor can be an emergency first responder (law enforcement, fire, and/or ambulance), a crime investigation organization such as the FBI, public safety personnel, or a private security firm (such as hired for security at a sporting game like the Olympics, Super Bowl, or World Cup). The action taken by the actor can include dispatching one or more first responder(s), such as a fire truck, an ambulance, or a police vehicle and associated first responder personnel, or causing an amber alert to be issued, for example.

Complex event modeler70is the analytic engine inside the server20that allows thousands to millions of data feeds to come in from the EPDs and then alert on pre-defined thresholds. For example, if a fire is seen in a video the complex event modeler70will send an alert to the nearest fire department and send notices to EPD users in the immediate area. In one embodiment, complex event modeler70may include the GeoVigilance tool commercially available from Transvoyant.

Predictive modeling tool80is the analytic engine inside the server20that takes the alerts and data from the complex event modeler70and then “predicts” the next likely group of scenarios. For example, a fire on or near a major thoroughfare would generate an alert that indicates traffic will cause delays in the arca and EPD users should plan accordingly. In one embodiment, predictive modeling tool80may include the SilverEye tool commercially available from Knowmadics, the Total Insight tool commercially available from Larus, or the Satellite Tool Kit (STK) commercially available from Analytical Graphics, Inc.

In one embodiment, CAD tool90includes the SilverEye web-based software application commercially available from Knowmadics, Inc. running in a CAD center. SilverEye may be the device management software in the system10that all the EPDs running the app are connected to. SilverEye in a CAD center allows data from EPDs to be visualized securely and quickly without having to replace the current investment legacy hardware/software in the CAD center. For example, a computer in the CAD center that has internet connectivity can visualize/playback imagery, video, and audio data from EPDs running the app as the data is collected to supplement the traditional data collected from a 911 call-location, voice description, and identity (phone number).

Alerts may be generated by the server20or CAD tool90based on the data received from the EPDs. For example, using SilverEye an operator can set an alert(s) based on certain conditions/groups of conditions being met or exceeded-location, time, key words, weather, and/or temperature etc. When the conditions set by the operator are met, the CAD tool90automatically generates an alert-machine-to-machine (M2M) or changes a condition on another device. For example, a geographic boundary/geo-fence can be created and when say 10 or more objects enter the boundary after 6 PM. Alerts may be generated by the server20or based on the data received from the EPDs. What triggers an alert-using SilverEye an operator can set an alert(s) based on certain conditions/groups of conditions being met or exceeded-location, time, key words, weather, temperature etc. that when the conditions are met automatically generates an alert—machine-to-machine (M2M) or changes a condition on another device. For example, a geographic boundary/geo-fence can be created and when say 10 or more objects enter the boundary after 6 PM, an alert can automatically be generated to a security guard to go check out the area for suspicious activity. In this case, an alert can automatically be generated by CAD tool90and transmitted to a security guard100to go check out the area for suspicious activity.

In another embodiment, server20or CAD tool90may generate alerts to be sent to EPDs by linking multiple EPDs to other types of devices such as cameras, audio recorders, trackers, seismic sensors, etc. For example, a geographic boundary can be set on the SilverEye control software so that when an EPD connected to system10enters, leaves, passes-by, etc. the geographic boundary an alert is generated which will enable a third party camera to track the EPD remotely without any human in the loop. That camera data can then be sent automatically to another EPD connected to system10.

In an exemplary embodiment shown inFIG.2, EPD150is a smartphone capable of collecting all of the above described data, which runs an app to perform these functions. The app allows selectable wireless transmission of a known and/or anonymous user's geographic location coordinates, audio, video, voice, text, temperature, velocity, or altitude (or any other sensed data available on the EPD150) to server20.

A user who witnesses an event can create a report on EPD150to upload to the server20.FIG.3shows exemplary screen shots of EPD150creating a report. Screen310shows an opening menu for creating a report. Screen320allows a user to select a type of report (police, fire, weather, lost child, etc.). Screen330allows the user to remain anonymous, and add whatever type of data they have collected to the report. The report can include a text summary of the incident the user wishes to report, and audio/video/photo attachments. The user identifies the type of alert, and the report, attachments are uploaded to the remote system, with the option to retain a copy of the report or to transmit without storing any data on the user's EPD. Multiple users (the crowd) witnessing the same incident/event can upload reports and sensor data about the event to the same remote system in a crowd-sourcing model. Data previously collected on the EPD150by other existing apps can be added to a CASES/AGENT report. For example, an image that was taken on an iPhone with the iOS® Camera app can be appended to a CASES report which is sent to a CAD tool90.

As shown inFIG.4, screen410shows alerts that other users in the vicinity of the event, and who have authorized their EPDs to receive alerts, can receive from the remote system20about the ongoing event. Screen420allows the notified user to provide further data to server20. Screen430allows the user to notify the server20of the user's own skillset or competency (e.g., law enforcement, firefighting, social work, medical training, search and rescue, crisis housing), and if the event calls for a particular competency, the remote system can automatically send alerts to all users with competencies relevant to the event and who are located in the vicinity of the event information about the event so that the user can utilize their competencies to ameliorate negative consequences caused by the event. Such users would have authorized the app to track their location so that the remote system can send alerts only to those users located close to the event of interest, regardless of whether those users have witnessed the event or submitted a report.

The server20can also provide a list of previously submitted reports to the EPD150. As shown inFIG.5, the EPD can display a map540with indicators550at each report location. Menu buttons510-530allow the user to select whether the map will include indicators to show their own location, other agents, and/or the report locations (“cases”). In the embodiment shown inFIG.5, the user's own location is being shown, with the map roughly centered on the user's location. Even if the user's location is turned off with button510, the map may still be centered on the user's location as a default state. The map shown inFIG.5is a street map, but any local map is within the scope of the invention, such as maps of stadiums as discussed below.

Further, the EPD can support a chat function which allows the EPD user to chat as shown inFIG.6. The EPD can display the distance630and direction610of a plurality of chat participants620so that the user can directly gather further information about local events, or warn others.

FIG.7shows that the EPD displaying a list of local cases, along with the direction710and distance720to the location of the case. When a user selects a particular case, further information730is provided. This further information may include some or all of the data the reporting EPD provided to the server20. This further information may be displayed on the map proximate the location of the case, as shown inFIG.8.FIG.8illustrates an exemplary map810with case location markers820and case information830.

Any user can also authorize the EPD to turn any selected sensor on the EPD on or off (e.g., microphone, camera, GPS, accelerometer) and upload the selected sensor outputs in real time to the server20. Further, by selecting the boss mode button920shown on screen910ofFIG.9, this can be done surreptitiously for the safety of the user. In this case, an innocuous screen is displayed during data collection, such as the exemplary game display1000shown inFIG.10. Any screen unrelated to data collection may be used to prevent a hostile person from seeing that the user is collecting and reporting data, possibly related to a crime being committed by the hostile person.

In another embodiment, server20issues a command to the EPD150to enter boss mode without any command by the user of EPD150. In this regard, a rule set can be established by server20based on conditions being met that would automatically enable collection to occur on the EPD150without the user having to do anything. In one embodiment, server20can command each EPD150to start recording/streaming video whenever the EPD150was within 1 mile of a landmark such as the Washington Monument. In another embodiment, the server20may command every EPD150within a set distance of a reported case to begin recording sensor data and to transmit the sensor data to server20.

The EPD also allows the user to select three levels of participation: anonymous in which the user uploads reports or sensor data anonymously, passive in which the user's personal identification information is reported with the sensor data uploaded, and remote control in which the user allows the remote system to control one or more sensors on the user's EPD for transmission to the remote system. The EPD can be placed in an invisible or surreptitious mode in which it will transmit sensor data in the background without conveying any human-discernible cues that it is doing so. In this regard,FIG.11shows that a server1110can send a command to device1120to collect and transmit data without the user knowing. The data is sent to repository1130to be analyzed by server1110.

FIG.12shows an exemplary display of the CAD tool90, which may be embodied by SilverEye™ software. The features of this tool are described above with respect toFIG.1.

In another exemplary embodiment, the above described features may be divided between two apps, the CASES app and the CASES AGENT app. CASES and CASES AGENT apps are distributed on EPDs with back end support provided through a cloud model controlled via an enterprise service bus (ESB). The primary CASES app turns EPDs into sensors and those sensors can be used in a crowd-sourced fashion to help law enforcement, public safety, and defense personnel in a time of crisis or danger. The CASES ecosystem involves the software app and the software back end data transformation which occurs in the cloud as data from the EPDs is analyzed and in the cloud. The CASES AGENT app has secondary features that allow it to be used (turned on and off) remotely.

Primary features include:

Philosophy of CASES and CASES AGENT=Collection-Transformation-Action

CASES and CASES AGENT are part of an ecosystem that includes a downloadable app which connects to a cloud based transformation engine which then sends machine to machine (M2M) and/or machine to person (M2P) alerts which cause action to occur in the real world.

It was designed primarily for everyday use as well as venue/event specific use. End users (civilians) can see an event and send data as quickly and easily as possible as an enhancement to public safety.

Data is received by the CASES back end processing capability in the cloud transforms the raw data feeds into a case.

The app can be customized by end users and white labeled for specific events-such as the Superbowl, Olympics, World Cup, Grand Prix, etc. In those instances the actual seating chart of the venue could be downloaded as an add on and users in the ecosystem can identify where they are sitting/standing so that when an event occurs the data they generate can be tied to a specific area within the event.

Uses the off-the-shelf/out-of-the-box capability of the EPDs to send data-location, audio, video, text, temperature, speed, altitude and any other data that can be collected by the EPD to the cloud for follow on analysis, cataloguing, and distribution.

Quick way for average citizen to share observations from their EPDs.

Venue Specific downloads are available so that at an event CASES users can let people know where they were sitting/standing etc. when an event occurred.

Everyday CASES users can register any particular skill set they have that would make them more useful in an actual emergency so that officials would know what type of Good Samaritan support there was near an emergency.

CASES reports can be shared with public safety and law enforcement personnel.

Directly to Law Enforcement, public safety, or to a “Cut out” server which is accessible by personnel at a computer aided dispatch (CAD) center or public-safety answering point (PSAP), sometimes called “public-safety access point” (a call center responsible for answering calls to an emergency telephone number for police, firefighting, and ambulance services). This CAD center may house the CAD tool90as described above.

An enhanced version of CASES called CASES AGENT has all of the same capability plus listed above plus:

The AGENT version can be remotely controlled by command and control (C2) software in server20to turn on/off the camera, audio and locational data streams from EPD150the AGENT version is hosted on, as shown inFIG.11. This command to enter boss mode by the server20does not involve any input by the user of EPD150, as discussed above.

AGENT Version has a panic button feature.

AGENT Version has a manually selected boss mode so that a user can make it appear as if the app is not running if they had to turn their EPD over for forensic inspection, as shown inFIG.10.

AGENT Version has a primary mission of information collection for public safety.

AGENT Version can be scheduled to turn and off based on time of day and/or location.

The remote system can communicate with, for example, the FBI, the DEA, other law enforcement, public safety, or military operations.

Thus, the CASES and CASES AGENT app technologies combine crowd sourcing with civic responsibility to create an ecosystem where modern technology-specifically the billions of dollars of investment in EPDs and the cloud—can be used to do good. It puts technology that is already in the hands of ordinary citizens to work for the common good. Some advantages of the CASES and CASES AGENT app include that it creates a central application to process and fuse multiple types of data from EPDs and then easily send it from the EPD to the cloud with a simple buttons.

The CASES and CASES AGENT apps are designed to be customized so that it can be licensed to a sponsor who becomes the sponsor of the app being used at specific events such as the Olympics, etc. It can be customized so that certain EPD features can be turned on and turned off in countries where data collection of this type is prohibited.

Additional advantages of the invention may include (this list is not exhaustive):1. Single screen app interface—as opposed to 2-4 separate applications with multiple interfaces, such as having a separate app to track a phone, an app to take a picture, an app to record audio, an app to record a video, or an app to chat.2. Multiple date feeds from multiple EPD sensors—as opposed to a user experience where each screen can only handle one feed at a time.3. Crowd sourced data inputs from social media—as opposed to just getting one way notification alerts from a Rich Site Summary (RSS) feed or broadcast.4. Can be used as an information collection and transmission tool in real time—as opposed to collecting data and then sending it at a later date in response to an alert or after an event. For example, the Boston Marathon Bombing had thousands of people collecting images and video, but without any way to easily and rapidly transmit that data to public safety and law enforcement personnel. The FBI was forced to manually collect data from EPDs from witnesses and then fly that data from Boston, MA to FBI facilities in Quantico, VA.5. Open Application Programming Interface (API) and Software Development Kit (SDK) so that end customers can enhance and extend the software themselves—as opposed to a closed, proprietary system, or non-existent SDK or API that forces end users to pay the developing company to extend the capability.6. Secure data transmission using Triple Data Encryption Standard (DES) or Advanced Encryption Standard (AES) 128/256 encryption for communications between the EPDs150and the data server20.7. Multi-modal data transmission pathways from the EPD150where data can be transmitted from the EPD150through either commercial terrestrial telephony (2G, 3G, 4G, LTE, etc.), WiFi, and/or satellite communications pathways.

Applications for the aspects of the present disclosure include:1. Public safety2. Emergency response3. Crime prevention4. Law Enforcement5. Intelligence collection6. Military/law enforcement hostile forces tracking7. Military/law enforcement blue force (Agent/CI) tracking8. Military/law enforcement mission planning9. Military sensor planning10. Critical installation protection

Multiple applications can be used in parallel and then combined on the server20.

Another exemplary embodiment of the present invention is the SNEEZES application, shown inFIGS.13-16. SNEEZES is an acronym for Syndromic Notifications for Epidemics or Elevated Zone Events System. The SNEEZES app technology combines crowd sourcing with civic responsibility to create an ecosystem where modern technology-specifically the billions of dollars of investment in EPDs and the cloud—can be used to do good. It puts technology that is already in the hands of ordinary citizens to work for the common good. Advantages of the SNEEZES app include that it creates one central application to process and fuse multiple types of data from EPD and then easily send it from the EPD to the cloud with a simple buttons.

In this regard,FIG.13shows an exemplary SNEEZES embodiment including EPD1350collecting and transmitting data as described with respect to the previous embodiments. The SNEEZES system thus includes a data collection, transformation and action ecosystem that includes a) an app front end for data collection where the data is provided by end users voluntarily about their general health from EPD1350, b) a hosted cloud-based enterprise service bus (ESB) for data transformation1320, and c) a hosted web site for the publication of “heat maps”1390made from the transformed data from app users. The heat maps1390can be viewed and distributed to on desk top as well portable platforms (laptops, phones, and tablets). A heat map1390is a graphical representation of data where the individual values contained in a matrix are represented as colors. In one embodiment of SNEEZES, the heat maps1390represent instances of people reporting feeling unwell or well.

EPD1350includes a sensor such as a microphone, a camera, a keyboard for receiving text data that may be embodied by a touchscreen displaying the keyboard, a thermometer, and a global positioning system (GPS) sensor. Further, EPD150may include more than one sensor, or all of the sensors listed above. In general, each EPD150will include at least one sensor, a processor, memory, a transmitter for transmitting the data to ESB1320, and a receiver for receiving data from ESB1320. ESB1320also includes a processor, a memory, a transmitter for transmitting data to the EPD1350, and a receiver for receiving data from EPD1350.

People like to talk about their health. The primary SNEEZES app allows the public to report in near-real-time about their general health. That data is then collected in the cloud and transformed into heat maps1390. Those heat maps1390can then have additional data sources overlaid on top of them to create dynamic and static views of population centers and the general health of people around them.

Further, alerts1310may be sent by the ESB1320to the EPD1350. These alerts may include health information, location information, and may also include advertisements to pharmacies, drug store chains, event hosts, and/or tourist bureaus, based on how the user is feeling.

In another embodiment, an initial threshold can be set in a complex event modeling tool within ESB1320such that if more than 100 unique SNEEZES app users report flu symptoms within 50 miles of each other it will trigger the control system to do a web search of that area for reports of flu. If both conditions are met, ESB1320will send an alert to the EPD1350of all SNEEZES app users in the area to warn of increased possibility of the flu.

In still another embodiment, the SNEEZES app may have the ability to automatically transmit body temperature data off of EPDs1350which can record body temperature. If a person in a quarantine area uses a SNEEZES enabled EPD1350, it would allow that persons' body temperature to be automatically recorded and forwarded through the SNEEZES app to a complex event modeling tool within ESB1320and aggregated with other SNEEZES collected data, as well as other third party collected data, to generate alerts back to EPDs1350of SNEEZES App users, as well as the general public.

FIG.14shows a display1410and a menu1420to allow a user to report their health information. Screen1430shows that the user can remain anonymous, and may attach any of the data collected to their health report before sending to ESB1320.

As shown inFIG.15, the user can edit personal settings, which may include their own competencies that they can provide the ESB1320.

FIG.16shows one embodiment of a display of a heat map1390. InFIG.16, menu bar1610allows a user to designate a number from 1-10 to indicate their general health. Button1620allows the user to enter and post their body temperature. The heat map1630shows different colors based on health conditions in that locality. The user's location is at indicator1640. The user can post text related to their health condition using button1650, and the user can enter their heart rate using button1660.

Thus in one embodiment, a user of the EPD1350enters data about their own health into EPD1350using the interface described above. The EPD1350sends the data entered by the user about their health to the ESB1320. The ESB1320incorporates that data into heat map1390and transmits the updated heat map1390to EPD1350, which can then display it for the user. Accordingly, the user can receive near-real-time updated heat maps providing health data covering the mapped area.Philosophy of SNEEZES=Collection-Transformation-Action

SNEEZES is an ecosystem that includes a downloadable app which connects to a cloud based transformation engine which creates heat maps1390then sends machine to machine (M2M) and/or machine to person (M2P) alerts which cause action to occur in the real world.

It was designed primarily for “everyday use” for the public to contribute near-real-time experiential public health information to the larger public for multiple end user purposes, including:Get helpGet couponsGet travel informationOther

Data is received by the SNEEZES back end processing capability in the cloud transforms the raw data feeds into a SNEEZES heat map1390which then shows people their information in context and allows browsers of the data to see the general health and wellbeing of a population area prior to going there or for general situational awareness.

Additional RSS feed data for pollen count, heat index, health warnings, etc. would also be overlaid onto the SNEEZES heat maps1390to create a holistic public health snapshot informed by multiple sources including SNEEZES users. This enhanced level of syndromic situational awareness could prevent issues like asthma attacks in areas where heat, pollen, and other events may trigger an attack.

The app can be customized by end users and white labeled for specific events—Superbowl, Olympics, World Cup, Spring Break, Ski Season, Mardi Gras, etc.

People going on or hosting trips and/or to these venues could make use of the data as a way to show how healthy the area they are going to might be in relation to other parts of the country.

Uses the off-the-shelf/out-of-the-box capability of the EPD to send data-location, audio, video, text, temperature, heart rate, pulse O2, etc. and any other data that can be collected by the EPD to the cloud for follow on analysis, cataloguing, transformation and/or to generate a heat map.

Quick way for average citizen to share observations from their EPDs about their general health and to see the general health of other parts of the globe.

Venue-Specific downloads are available so that at an event SNEEZES users can let people know where they were sitting/standing etc. when an event occurred

SNEEZES reports can be shared with public safety and health officials either directly to them or to a “Cut out” server they have access to.

The apps are designed to be customized so that it can be licensed to say a corporation or tourist bureau so that they become the sponsor of the app being used at specific events such as the Olympics, etc. It can be customized so that certain EPD features can be turned and turned off in countries where data collection of this type is prohibited.

Other advantages may include:1. Single screen interface—as opposed to 2-4 separate applications with multiple interfaces.2. Multiple data feeds from multiple EPD sensors—as opposed to a user experience where each screen can only handle one feed at a time.3. Crowd sourced data inputs from social media—as opposed to just getting one way alerts from an RSS feed or broadcast.4. Can be used as a public health collection tool in real time—as opposed to collecting data and then sending it a later date in response to an alert.5. Open API and SDK so that end customers can enhance and extend the software themselves—as opposed to a closed, proprietary, or non-existent SDK or API that forces end users to pay the developing company to extend the capability.

Applications for SNEEZES include:1. Public health and cohort tracking2. Public safety3. Emergency response4. Intelligence collection5. Military/law enforcement health and cohort tracking6. Military sensor planning

The term cohort (as used above) effect is used in social science to describe variations in the characteristics of an area of study (such as the incidence of a characteristic or the age at onset) over time among individuals who are defined by some shared temporal experience or common life experience, such as year of birth, or year of exposure to radiation.

The system allows for using multiple applications in parallel and then combining on the server/cloud side.

The present written description uses examples to disclose the present subject matter, including the best mode, and also to enable any person skilled in the art to practice the present subject matter, including making and using any devices or systems and performing any incorporated and/or associated methods. While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.