Content presentation and augmentation system and method

A computerized system and method are presented that creates implicit content on a mobile device by monitoring and recording input from sensors on the device. Metadata from the implicit content and from user-created content is then analyzed the purpose of event identification. Using the metadata and event identification, the content is created into clusters, which can be confirmed by the user as actual events. Events can then be grouped according to metadata and event information into a presentation grouping. Presentation groupings can be presenting using a map and timeline interface. The system augments content concerning an event by searching for related content. Users viewing the presentation grouping can add additional user content. The system can also add content from users that attended the same event.

FIELD OF THE INVENTION

The present application relates to the field of computerized systems that organize and display media content.

SUMMARY

An embodiment of the present invention creates implicit content on a mobile device by monitoring and recording input from sensors on the device. This embodiment also analyzes metadata from the implicit content and metadata from explicit content created by a user for the purpose of creating content clusters, which are confirmed by the user as actual events. Events can then be grouped according to metadata and event information into a presentation grouping. Presentation groupings can be presented to a using an interface having a timeline, map, and content sections. Presentation groupings can include augmentation content, including external augmentation content taken from the Internet based on the location and time information in the displayed event. External augmentation can be stored with the event data and formatted according to user desires. Individuals viewing a presentation grouping can add additional augmentation that is stored with the event data. Furthermore, the system can automatically augment event data using data from other users of the system that participated in the same event.

DETAILED DESCRIPTION

System Overview

FIG. 1shows a mobile device100utilizing one embodiment of the present invention. The mobile device100can communicate over a wide area network170with a plurality of computing devices. InFIG. 1, the mobile device100communicates with a media organization server180, a global event database server190, one or more cloud content servers192, and a third-party information provider server194.

The mobile device100can take the form of a smart phone or tablet computer. As such, the device100will include a display110for displaying information to a user, a processor120for processing instructions and data for the device100, a memory130for storing processing instructions and data, and one or more user input interfaces142to allow the user to provide instructions and data to the mobile device100. The display110can be use LCD, OLED, or similar technology to provide a color display for the user. In some embodiments, the display110incorporates touchscreen capabilities so as to function as a user input interface142. The processor120can be a general purpose CPU, such as those provided by Intel Corporation (Mountain View, Calif.) or Advanced Micro Devices, Inc. (Sunnyvale, Calif.), or a mobile specific processor, such as those designed by ARM Holdings (Cambridge, UK). Mobile devices such as device100generally use specific operating systems designed for such devices, such as iOS from Apple Inc. (Cupertino, Calif.) or ANDROID OS from Google Inc. (Menlo Park, Calif.). The operating systems are stored on the memory130and are used by the processor120to provide a user interface for the display110and user input devices142, handle communications for the device100, and to manage applications (or apps) that are stored in the memory130. The memory130is shown inFIG. 1with two different types of apps, namely content creation apps132and a media organization app134. The content creation apps132are apps that create explicit media content136in the memory130, and include video creation apps, still image creation apps, and audio recording apps. The media organization app134creates implicit content138. The media organization app134is responsible for gathering the different types of explicit media content136and the implicit content138(referred to together as content140), analyzing the content140, and then organizing the content140into clusters, events, and presentation groupings that are stored in media organization data139as described below.

The mobile device100communicates over the network170through one of two network interfaces, namely a Wi-Fi network interface144and a cellular network interface146. The Wi-Fi network interface144connects the device100to a local wireless network that provides connection to the wide area network170. The Wi-Fi network interface144preferably connects via one of the Institute of Electrical and Electronics Engineers' (IEEE) 802.11 standards. In one embodiment, the local network is based on TCP/IP, and the Wi-Fi network interface includes TCP/IP protocol stacks. The cellular network interface146communicates over a cellular data network. The provider of the cellular data network then provides an interface to the wide area network170. In one embodiment, the wide area network170is the Internet.

The mobile device100uses sensors150for a variety of purposes on the device100. In the present embodiment, the sensors150provide the means to create media content136. The content creation apps132respond to signals from the user input142to capture media content136using the camera sensor152and the microphone154. These types of media content136are known as “explicit media content” because the user has explicitly requested that the mobile device100capture and store this media content136. For instance, a user might instruct a photo taking app132to take a still photograph using the camera152, or to stitch together a stream of input from the camera sensor152into a panorama image that is stored as explicit media content136. A movie app132might record input from the camera152and microphone154sensors as a video file136. Or a voice memo app132might record input from the microphone sensor154to create an audio media content file136. In each case, these content creation apps132respond to an explicit request from a user to create the media content136. In most cases, the explicit media content136is stored as a file or a data record in the memory130of the mobile device100. This file or data record includes both the actual content recorded by the sensors150and metadata associated with that recording. The metadata will include the date and time at which the media content136was recorded, as determined by the clock156. Frequently, the metadata also includes a geographic location where the media content136was created. The geographic location can be determined from the GPS sensor158, or by using other location identifying techniques such as identifying nearby Wi-Fi networks using the Wi-Fi Network Interface144, or through nearby cell tower identification using the cellular network interface146. Some content creation apps132will include facial recognition capabilities in order to tag the identity of individuals within a photo or video file136. Other apps132will allow a user a manually tag their files136so as to identify the individuals (or “participants”) portrayed in those media files136. These identity tags can then be added to the metadata stored with the media content file136in memory130.

In some embodiments, the explicit media content136will be stored remotely on a cloud content server192. For example, all photographs taken by the camera152may be stored in memory130as explicit media content136and may also be transmitted over one of the network interfaces144,146to the cloud content server192. The locally stored explicit media content136may be temporary in nature, with permanent storage provided on the cloud content server192. In some circumstances, the cloud content server192will be provided by a third party, such as the FLICKR service provided by Yahoo! Inc. of Sunnyvale, Calif.

The media organization app134creates implicit content138by monitoring the sensors150on the mobile device100and storing related data as implicit content138when it monitors an interesting change in the sensors150. For instance, the media organization app134might be monitoring the GPS sensor158and accelerometer160during a family driving vacation from Chicago, Ill. to Yellowstone National Park in Wyoming. The accelerometer160can indicate when the family car stops, and then determine the location of the mobile device100using the GPS sensor158. By monitoring the accelerometer160and the GPS sensor158(at least periodically), the media organization app134can determine that the car was stopped during this family vacation for 3 hours, 15 minutes in Wall, S. Dak. This data could be stored as implicit content138in the memory130.

When the app134creates this implicit content138, it may also uses one of the network interfaces144,146to obtain additional information about this implicit content138. For example, the app134may contact a global event database server190that contains information about a great number of events (or “occurrences”). This type of database server190, which is provided by several third parties over the Internet170, allows users to specify a geographic location and a time, and the server190will respond with information about occurrences happening near that location around that time. The information returned from the global event database server will generally include a title for the occurrence, a description for that occurrence, a time period during which that occurrence takes place, and an exact physical location for that occurrence. For example, during the stop in Wall, S. Dak., the app134may inquire whether there are any events happening in Wall at the time the vehicle was stopped. The event database server190may indicate that at this time, a parade was happening in downtown Wall. The app134may also make inquiries from different information provider servers194, such as a server194that provides weather information for a particular geographic location. By acquiring this information from external database sources190,194, the media organization app134would be able to create implicit content138indicating that from 12:15 to 3:30 pm on Jul. 4, 2013, the user of the mobile device100stopped in Wall, S. Dak. and witnessed a parade in sunny, 92 degree weather.

The media organization app134can take advantage of any of the sensors150on the mobile device100, including the camera152, microphone154, clock156, GPS sensor158, accelerometer160, gyroscope162, ambient light sensor164, and proximity sensor166. The app134can define monitoring modes that determine the extent to which it monitors the various sensor150. For instance, in one monitoring mode the app134could provide reverse geocoding by periodically (or continually) recording a location for the user from the GPS sensor158. In another mode, the app134could monitor the accelerometer to indicate when the user is moving or has stopped moving. In a third mode, the app134could periodically monitor the microphone154. If no interesting noises are detected, the app134would wait for the next interval before it again monitored the microphone154. If interesting noises were detected (e.g., noises that were characteristic of human voices), the app134could record a small amount of the conversation and record it as implicit content138in memory130, along with the time and location at which the conversation was recorded. In a fourth mode, the use of another app, such as one of the content creation apps132, triggers the creation of an implicit content file138. For instance, the use of a photo or movie app132may cause the media organization app134to record the GPS location, the current weather, and the current event, if any, noted by the global event database server190. In addition, the app132in this fourth mode may record sounds from the microphone154to capture conversations between the user of the mobile device100and her photography subjects. These conversations would be stored as implicit content138in memory130.

When requested by the user, the media organization app134collects the content140from the memory130(and from cloud content servers192) and organizes the content140into content clusters. Content clusters are groups of content140that are grouped together as belonging to a particular occurrence or event. As described below, content clusters are presented to the user for modification and verification, after which the content groupings are referred to as user-verified events. Events may involve numerous elements of content140, or may involve only a single element of content140. In the preferred embodiment, the content clusters and events are stored in media organization data139. In addition, the content clusters and events could be stored on a media organization server180accessible by the mobile device100over the network170.

The media organization server180contains a programmable digital processor182, such as a general purpose CPU manufactured by Intel Corporation (Mountain View, Calif.) or Advanced Micro Devices, Inc. (Sunnyvale, Calif.). The server180further contains a wireless or wired network interface184to communicate with remote computing devices, such as mobile device100, over the network170. The processor182is programmed using a set of software instructions stored on a non-volatile, non-transitory, computer readable medium186, such as a hard drive or flash memory device. The software typically includes operating system software, such as LINUX (available from multiple companies under open source licensing terms) or WINDOWS (available from Microsoft Corporation of Redmond, Wash.).

The processor182performs the media organization functions of server180under the direction of application programming187. Each user of the server180is separately defined and identified in the user data188. The media organization app134can assist the user in creating an account on the media organization server180. The account can require a username and password to access user content189that is stored on the server180on behalf of the users identified in data188. The media organization server180can operate behind the media organization app134, meaning that the user of the mobile device100need only access the server180through the user interface provided by the app134. In addition, the media organization server180can provide a web-based interface to the user content189, allowing a user to access and manipulate the user content189on any computing device with web access to the Internet170. This allows users to organize their user content189and format presentations of that data189via any web browser.

Because the media organization server180contains information about content clusters and events created by a number of users, this server180can easily create its own database of past occurrences and events that could be useful to the media organization app134when clustering media. For instance, a first user could cluster media about a parade that they witnessed between 12:30 and 1:30 pm in Wall, S. Dak. on Jul. 4, 2013. The user could verify this cluster as a user-verified event, and could add a title and description to the event. This data would then be uploaded to the user data188on server180. At a later time, a mobile device100of a second user could make an inquiry to the media organization server180about events that occurred in downtown Wall, S. Dak. at 1 pm on Jul. 4, 2013. The server180could identify this time and location using the event created by the previous user, and return the title and description of the event to the mobile device100of the second user. In effect, the media organization server180could become a crowd-sourced event database server providing information similar to that provided by server190(except likely limited to past and not future events).

Content Handling

FIG. 2schematically illustrates the interaction of the media organization app134with content140and the other inputs that allow the media organization app134to create content clusters. In one embodiment, the content140is found in the physical memory130of the mobile device100. In another embodiment, this data140is found on “the cloud”200, meaning that the data is stored on remote servers180,192accessible by the mobile device100over network170. The dual possible locations for this content140is shown inFIG. 2by locating the data140both within memory box130and the dotted cloud storage box200.

The explicit media content136shown inFIG. 2includes video content222, photo content232, and audio content242. The video content222is created by a video app220running on the processor120of the mobile device100. When the video content222is created, it is stored along with metadata224that describes the video content222, including such information as when and where the video was created. Similarly a photo app230creates the photo content232and its related metadata234, and a voice recording app240creates audio content242and metadata244. These three apps220,230,240may be standard apps provided along with the mobile operating system when the user purchased the mobile device100. The data222,232,242from these apps220,230,240are stored in known locations in the local memory130or on the cloud data system200.

Third party or specialty apps250,260can also create explicit content136that is accessed by the media organization app134. The first specialty app250creates both photo content232and audio content242, and stores this data232,242and related metadata234,244in the same locations in memory130where the standard apps230,240provided with the device100store similar data. The second specialty app260also creates explicit media content262and related metadata264, but this content262is not stored in the standard locations in memory130. However, as long as the media organization app134is informed of the location of this specialty app content262on memory130, such content262can also be organized by the app134.

In addition to the explicit content222-262, the media organization app134also organizes implicit content138and its metadata274. In one embodiment, this implicit content138is created by the same app134that organizes the content140into content clusters. In other embodiments, the media organization app134is split into two separate apps, with one app monitoring the sensors150and creating implicit content138, and the other app134being responsible for organizing content140.

FIG. 2also shows a calendar app210creating calendar data212on the mobile device100. In one embodiment, this data can be used by the media organization app134as it arranges content140into content clusters. As explained below, the calendar data212may have explicit descriptions describing where the user was scheduled to be at a particular time. The media organization app134can use this data to develop a better understanding about how to organize content140that was acquired at that same time. The app134also receives additional information about occurrences and events from the global event database server190and the crowd-sourced event data from the media organization server180. The data from these sources180,190is also very useful to the app134as it organizes the content140.

The app134accesses all this content140, from the same locations in which the data was originally stored by the creating apps210-260and organizes it into content clusters using additional data from servers180and190. In most cases, the content140is organized based primarily on the metadata224,234,244,254,264, and274that was attached to the content140by the app that created the content140. In some circumstances, the media organization app134can augment the metadata. For instance, the app134could use facial recognition (or voice recognition) data280available on the mobile device100or over the network170to identify participants in the content140. Such recognition can occur using the processor120of the mobile device, but in most cases it is more efficient to use the processing power of a cloud content server192or the media organization server180to perform this recognition. Regardless of where it occurs, any matches to known participants will be used by the app134to organize the content140.

Example Content Clusters, Events, and Presentation Grouping

FIG. 3shows an example of one embodiment of a media organization app300organizing a plurality of items310-370into two content clusters380,390. In this case, there are three items of explicit content, namely content one310, content two320and content three330. Content one310is associated with three items of metadata312-316, which indicate that content one310was acquired at time “Time1” (312), at location “Loc.1” (314), and that participants A and B participate in this content (metadata316). Content one310could be, for example, a photograph of A & B, taken at Time1and Loc.1. Similarly, the metadata322-326for content two320indicates that it was acquired at time “Time1.2” (slightly later than time “Time1”), location “Loc.1.1” (close to but not the same as “Loc.1”), and included participants A & C. The metadata for content three330indicates only that it occurred at time “Time2.1”.

In addition to the three explicit content items310,320,330, the media organization app300is also organizing one implicit content item340, which has metadata indicating that it was taken at time “Time2” and location “Loc.1”. The media organization app300has also obtained data350from one of the event database servers180,190. This data350indicates (through metadata352-356) that an event with a description of “Descr.1” occurred at location “Loc.1” for the duration of “Time1-1.2”. Finally, the app300pulled relevant information form the calendar data212and discovered two relevant calendar events. The first calendar item360indicates that the user was to be at an event with a title of “Title1” at time “Time1”, while the second calendar item370describes an event with a title of “Title1” at time “Time2”.

The media organization app300gathers all of this information310-370together and attempts to organize the information310-370into content clusters. In this case, the app300identified a first cluster380consisting of explicit content one310, explicit content two320, event database information350, and calendar item one360. The media organization app300grouped these items of data310,320,350,360primarily using time and location information. The app300recognized that each of these items occurred at a similar time between “Time1” and “Time1.2”. Furthermore, to the extent that the items310,320,350,360identified a location, the location was either “Loc.1” or close by location “Loc.1.1”. One advantage of using calendar data212or data from event databases180,190is that some of this data212,180,190will identify not just a single time but an actual time duration. For instance, the calendar data212may indicate that a party was scheduled from 6 pm to 2 am. Based on this duration information, the media organization app300will be more likely to cluster content from 6 pm and content at 1 am as part of the same event. Similarly, the calendar data212may identify a family camping trip that lasts for two days and three nights, which might cause the app300to group all content from that duration as a single event.

Once the media organization app300identifies items310,320,350,360as being part of the cluster380, it stores this information in media organization data139on the mobile device100. This information may also be stored in the user content189stored for the user on the media organization server180. The information about cluster380not only identifies items of data310,320,350,360, as belonging to the cluster, but also aggregates the metadata from these items into metadata382for the entire content cluster380. This metadata382includes metadata from the explicit content310-320, which indicated that this content within this cluster380occurred during the time duration of “Time1-1.2” and at location “Loc.1.” The metadata from content310and320also indicated that this content involved participants A, B, and C. In addition, because the media organization app300accessed the calendar data212and the data from the event database servers180,190, the content cluster metadata282can also indicate that this content relates to an event with the title “Title1” having a description “Descr.1”.

The second content cluster390grouped together explicit content330, implicit content340, and calendar item two370primarily because these items330,340,370all occurred at time “Time2” or soon thereafter (“Time2.1”) and indicated either that they occurred at the same location (“Loc.1”) or did not indication a location at all. The cluster metadata392for this content cluster390indicates the time frame (“Time2-2.1”) and location (“Loc.1”) taken from the explicit content330and the implicit content340. The metadata392also includes the title “Title1” from calendar item2, which was linked with the others items330,340by the common time frame.

An important feature of this embodiment of the present invention is that the clustering of content380,390is done automatically without user involvement. The user only needs to create explicit content136with their mobile device100using their normal content creation apps132. These apps132save their explicit content136as usual. The media organization app300can run in the background creating implicit content138(pursuant to earlier user instructions or preference settings). At a later time, the media organization app300gathers the content140, makes inquiries from external event databases180,190, examines the user calendar data212, and then creates content clusters280,290for the user. This later time can be when the media organization app300is opened by the user and the user requests that the content clustering step occur. Alternatively, this later time can occur periodically in the background. For instance, the user may request through preference settings that the content clustering and database inquiries take place every night between midnight and two a.m., but only when the mobile device100is plugged into a power source.

Because the content clustering shown inFIG. 2takes place without user involvement, the media organization app300preferably gives the user the right to affirm or correct these clusters380,390. InFIG. 4, content cluster one380, cluster two390, and a third content cluster410are presented to a user through a user interface, represented inFIG. 4by element400. The user interface400presents these clusters380,390,410and their contents for the user to review. The user can confirm a cluster as accurate and complete, as this user did with content cluster one380. When a cluster380is confirmed, the media organization app300will consider the cluster to be a user-confirmed event, such as event one420shown inFIG. 4. Note that event one420contains the same metadata382that the content cluster380had before it was confirmed

Sometimes the user will wish to consolidate two different clusters into a single event. InFIG. 4, the media organization app300created separate clusters390,410, with cluster Two390occurring at time “Time2” and cluster three410occurring at time “Time2.5.” While the app300viewed these time frames as different enough as to create two separate clusters390,410, the user inFIG. 4chose to combine the separate clusters390,410into a single user-confirmed event two430. Note that the metadata432for event two430includes a time frame “Time2-2.5” derived from the metadata392,412of both of the original content clusters390,410. The event two metadata432also can contain user added additions, such as the user description433of this event430.

Each user-defined event includes one or more content items140that relate to a particular event that was likely attended by the user. The event might be a wedding, a party with a friend, or a child's swim meet. By clustering the content140together into events420,430, the user can better appreciate the content140. Furthermore, these events420,430are enhanced by the addition of implicit content138, and by the added data from calendar data212or one of the event databases180,190.

InFIG. 5, the media organization app300is being used to establish a presentation grouping500. A presentation grouping500is a grouping of two or more events according to a common subject for presentation together. The presentation may be slide show, a video file, a web site, or some unique combination that combines the media from multiple events420,430into a single presentation. Events420,430are grouped together by a common theme or subject. It is possible that some events420,430will be grouped into multiple presentation groupings500, while other events will not be grouped into any presentation groupings500.

InFIG. 5, event one420is shown having title “Title1” taken from the calendar item one360and event two430also has a title of “Title1” taken from calendar item two370. The media organization app300recognizes this commonality, and then suggests that these two events420,430be combined into a single presentation grouping500. This grouping500contains both events420,430, and has metadata502taken from the metadata422,432of the two events420,430. InFIG. 5, metadata502that was shared by all events420,430in the presentation grouping500are bolded (namely the timeframe “Time1-2.5”, the location “Loc.1” and the title “Title1”), which indicates that these elements in the metadata502are most likely to apply to the presentation grouping as a whole500.

Frequently, many events will be combined into a single presentation grouping500. For instance, a user may have ten calendar entries all labeled “Third Grade Swim Meet.” Although this parent attended all ten swim meets, the parent took pictures (i.e., created explicit media content136) at only six of these meets. The media organization app300will cluster this content136into six content clusters, with each cluster also containing a calendar entry with the same “Third Grade Swim Meet” title. Because of this commonality, the app300will automatically create a presentation grouping500containing content136from all six swim meets without including intervening content that is not related to the swim meets.

It is true that, in the example shown inFIG. 5, these two events420,430may not have been grouped in a single presentation grouping500if the user had not created calendar entries with the same title “Title1” for each event. While they shared the same location (“Loc.1”), this might not have been enough commonality for the app300to group the events420,430together. However, if these events were swim meets and were sponsored by an organization that posted every meet in the global event database server190, this presentation grouping500could still be created. As long as one item in a cluster identifies a location and another identifies a time, then the global event database server190should be able to identify any events were scheduled at the same location and time. Each event420,430would then include the identification of the event received from the global event server190, and the media organization app300would be able to group the same events420,430as a presentation grouping500.

Alternatively, another parent of a child in the third grade swim team may have created and labeled events using the media organization app300. When this data was uploaded to the media organization server180, the server180would now have knowledge of these swim meets. When the next user attempts to cluster content taken at the same swim meets, the media organization app300would query the server180and receive an identification of these swim meets, which would be added into their own events420,430.

Clustering Methods

FIG. 6shows a method600that is used to create implicit content138on the mobile device100. The method begins at step610, during which a user selects a particular mode to be used to monitor the sensors150of the mobile device100. The selected monitoring mode establishes which of the sensors150will be monitored by the method600, and also establishes a trigger that will be use to start recording data. For example, a walking tour mode could be established in which an accelerometer is routinely (every few seconds) measured to determine whether an individual is currently walking (or running). A trigger event could be defined to detect a change in the walking status of the individual (e.g., a user who was walking is now standing still, or vice versa). Alternatively, the trigger could require that the change in status last more than two minutes. This alternative walking tour mode would be designed to record when the user starts walking or stops walking, but would not record temporary stops (for less than two minutes). So a user that is walking down a path may meet a friend and talk for ten minutes, and then continue down the path. When the user reaches a restaurant, the user stops, has lunch, and then returns home. This mode would record when the user started walking, when the user stopped to talk to a friend, when the user started again, when the user ate lunch, when the user finished lunch and stared walking home, and when the user returned home. This mode would not record when the user stopped to get a drink of water (because the user stopped for less than two minutes), or when the user got up at lunch to wash his hands (because the user walked for less than two minutes). Other modes might include a car trip mode, which would monitor an accelerometer and GPS device to record car stops that lasted longer than an hour, or a lunch conversation mode, which randomly monitors the microphone to listen for human voices and records one minute of the conversation if voices are recognized. The point of selecting a monitoring mode in step610is to ensure that the user approves of the monitoring of the sensors150that must be done to create implicit content138, and that the user desires to create this type of content138.

Once the mode is established, the processor120will monitor the sensors150of the mobile device100at step620looking for a triggering event. The sensors150to be monitored and the triggering event will be determined by the selected monitoring mode. If the processor120detects a trigger at step630, the processor120will record data from the sensors150in step640. Note that the data recorded from the sensors150does not have to be limited to, or even include, the sensor data that was used to detect the trigger in step630. For instance, the triggering event may be that the user took their cellular phone100out of their pocket. This could be determined by monitoring the accelerometer160and the ambient light sensor164. When this occurs, the processor120might record the location of the device100as indicated by the GPS sensor158, the current time as indicated by the clock156, and the next two minutes of conversation as received by the microphone154.

Step650determines whether data from external sources are to be included as part of this implicit content138. Such data may include, for example, the weather at the currently location of the device100, or the presence of mobile devices100belonging to friends in the general proximity. If step650determines that external data will be included, a request for external data is made in step652, and the results of that request are received in step654. For example, the media organization app134might request local weather information from another app on the mobile device100or from a weather database194accessible over the network170. Alternative, a “locate my friends” app that detects the presence of mobile devices belong to a user's friend could be requested to identify any friends that are nearby at this time. The data from these apps or remote servers is received at step654, and combined with the data recorded from the sensors150at step640.

At step660, a determination is made whether to save this accumulated data. In some circumstances, a monitoring mode may establish that the data gathered after a triggering event (step630) is always to be stored as an implicit content138. In other circumstances, the monitoring mode may impose requirements before the data can be saved. For instance, the lunch conversation mode may not save the recorded audio as implicit content138if analysis of the recording indicates that the voices would be too muffled to be understood. If the condition for saving the data under the monitoring mode is met at step660, then the data (including both sensor data recorded at step640and external data received at step654) is recorded as implicit content at670. If the step660determines that the condition is not met, step270is skipped. At step680, the process600either returns to monitoring the device sensors150at step620, or ends depending on whether additional monitoring is expected by the monitoring mode.

FIG. 7shows a method700for clustering content140into content clusters. The process700starts at step705by gathering the explicit content136from the memory130on the mobile device100, a cloud storage server192, or both. Next the implicit content138is gathered at step710, again either from memory130or from user content storage189at server180. These steps705,710may gather all information available at these data locations, or may only search for new content140added since the last time the app134organized the content140.

At step715, the media organization app134accessing facial or voice recognition data280in order to supplement the participant information found in the metadata for the gathered content140. Of course, this step715could be skipped if participant information was already adequately found in the metadata for the content140, or if no participant recognition data280were available to the app134.

At step720, the media organization app134analyses the metadata for the content140, paying particular attention to location, time, participant, and title metadata (if available) for the content140. Using the time information taken from the content140, the app134analyzes the calendar data212looking for any calendar defined events that relate to the content140being analyzed (step725). In addition, the app134uses time and location information from the content140to search for occurrence information from one or more third party event databases190(step730). The app134also makes a similar query at step735to the crowd-sourced event definitions maintained by the media organization server180. If the calendar data or the responses to the queries made in steps730,735contain data that is relevant to the content140being analyzed, such data will be included with the content140at step740.

At step745, the content140and the relevant data from steps725-735are clustered together by comparing metadata from the content140and the added data. In one embodiment, clusters are based primarily on similarities in time metadata. In this embodiment, the app134attempts to group the content140by looking for clusters in the time metadata. In other embodiments, location metadata is also examined, whereby the app134ensures that no content cluster contains data from disparate locations.

At step750, metadata is created for the content clusters by examining the metadata from the content140and the additional data obtained through steps725-735. The clusters are then stored in the media organization data139in memory130, in the user content189of the media organization server180, or both.

At step760, the automatically created content clusters are presented through a user interface to a user for confirmation as user-confirmed events. The user can confirm a cluster without change as an event, can split one cluster into multiple events, or combine two or more clusters into a single event. The app134receives the verified events from the user interface at step765. The user can also confirm and supplement the metadata, adding descriptions and tags to the metadata as the user sees fit. Finally, the verified events are saved in step770with the media organization data139in memory130, and/or in the user content189of the media organization server180. As explained above, these data locations139,189can be designed to hold only the organizational information for the content140while the content140itself remains in its original locations unaltered. Alternatively, all of the organized content140can be gathered and stored together as user content189stored at the media organization server180. While this would involve a large amount of data transfer, the media organization app134can be programmed to upload this data only in certain environments, such as when connected to a power supply, with access to the Internet170via Wi-Fi Network Interface144, and only between the hours of midnight and 5 am. Alternatively, this data could be uploaded continuously to the remote media organization server180in the background while the mobile device100is otherwise inactive or even while the device100is performing other tasks.

FIG. 8shows a method800for grouping events into presentation groupings. This method800starts at step805, wherein events are identified by the media organization app134for grouping. Step805might be limited to clusters that have formally become user-verified events through steps765and770. Alternatively, the process800may include unverified content clusters stored at step755. At step810, the app134examines the metadata for each event and cluster, and then attempts to find commonalities between the events and clusters. As explained above, these commonalities can frequently be based upon event information obtained from calendar data212or from data obtained by outside event data180,190.

In one embodiment, step810uses commonality in the metadata that does not relate to closeness-in-time. The reason for this is that content that was collected close to the same time as other similar content would, in most cases, have already been clustered together into events. Consequently, it is likely that the separate events being grouped together into a presentation grouping would not share a common time with one another. However, it may be useful to recognize commonalities in the time metadata that are not related to closeness-in-time. For instance, the app134may recognize that numerous content clusters or events occur on Thursday evenings from 6 pm to 8 pm. The app134may recognize this as a connection between the events, and therefore propose combining all events that occur on Thursday evenings from 6 pm to 8 pm as part of a presentation grouping.

At step815, the app134uses the metadata from the combined events to create metadata for the presentation groupings. The presentation groupings and metadata are then stored at step820in the media organization data139or in the user data189on server180.

At step820, the user is allowed to verify the presentation groupings created at step810. The user is given the ability to add events or content140directly to a presentation grouping, or to remove events or content140from the presentation grouping. The user is also given the ability to modify the metadata, and to format the presentation grouping as desired by the user. As explained above, the presentation grouping may be used to create a web site, a slide show, or a video presentation of the combined content. As a result, numerous formatting options will be available to a user at step825to format the presentation grouping. At step830, the user modifications to the presentation groupings are stored at locations139or189, and the process800ends.

Presentation and Augmentation

FIG. 9shows a sample presentation grouping900. The metadata910for this presentation grouping900shows that the events920that were grouped together all related to a family's Yellowstone driving trip that took place from Jul. 2 to Jul. 14, 2012. This presentation grouping900includes events920that took place in Illinois, Wisconsin, Minnesota, South Dakota, and Wyoming.

The presentation grouping900could include tens or even hundreds of events920.FIG. 9shows details for only three events, event one930, two940, and three950. It should be understood that numerous events920might occur in before, between, or after these particular events930,940,950. Event one930took place on Jul. 2, 2012 and related to leaving the family home in Chicago, Ill. Event two940took place in the Badlands of South Dakota on the morning of Jul. 4, 2013. Event three950took place when the family watched the parade in Wall, S. Dak. on Jul. 4, 2013.

The events in presentation grouping900are used to create the user interface1000shown inFIG. 10. In the preferred embodiment, the user interface1000is created by the media organization server180. This is true even if the presentation grouping900was created on the mobile device100using the media organization app134. In one embodiment, when the presentation grouping900is created and stored in the media organization data139, it is also uploaded via network170to the user content189stored on the media organization server180. Alternatively, the content within the events920could have been uploaded to the server180, and the server180could have assisted the user in creating the presentation grouping900. One benefit to having the media organization server180create interface1000is that the interface1000could then be accessed by any user with access to the Internet170. In one embodiment, the server184operates as a web server under the guidance of application programming187, and the interface1000is a web interface provided over the Internet170.

To create this interface1000, the server184analyzes all of the events920in the presentation grouping900, including the events one930, two940, and three950. The earliest and latest times associated with these events920are identified (in this case, Jul. 2, 2013 and Jul. 14, 2013. A graphical timeline is then created that spans between these dates. InFIG. 10, this timeline1020shows a portion of the entire timeline of the presentation grouping900. A user can see more of the timeline by selecting arrows1022and1024. The user could also “zoom out” to see all of the timeline1020using user interface element1032. The timeline1020includes a single time indicator, in this case a black circle1026. The user can drag this circle1026along the timeline1020to input a desired time. In this case, the user has located the circle1026on Jul. 4, 2013. More specifically, Jul. 4, 2013, between 11:00 and 11:15 am, as indicated by the heading1010shown on the interface1000. This time corresponds to event two940.

In addition to the timeline1020, the interface1000also includes a map1030. Maps1030are particularly useful elements when displaying a presentation grouping900that includes a plurality of different locations. For this family trip to Yellowstone, the map1030will show the family's route during the vacation. In this case, the map1030shows the states that the family traveled in during the time shown on the timeline1020(namely between Jul. 3 and Jul. 9, 2013). A user can zoom into or out of the map1030using interface element1032. In one embodiment, using interface element1032will simultaneous zoom both the timeline1020and the map1030, with the map1030always showing locations covered by the time period shown in the displayed portion of the timeline1020. In other embodiments, separate zoom interface elements1032will be provided for the timeline1020and the map1030.

The map1030includes a path1040showing the path of the user along the timeline. In this case, the path1040shows a route through Minnesota, South Dakota, and Wyoming and into Yellowstone National Park. The path1040can be derived from the events920within the presentation grouping900being displayed. In most cases, the presentation grouping900will not have explicit information about every point on the path1040, but instead will have multiple, discrete events920along the path1040. The points along the path1040that are associated with actual events920in the presentation grouping900are shown inFIG. 10with short line hash marks1042on the path1040. The portions of path1040that exists between these locations can be filled in using an intelligent “guess.” For instance, most of the events920occur on U.S. Interstate 90, so the server180can guess that the user traveled between these events920following the Interstate. Alternatively, the user can manually add the correct path1040between the locations of the events920. In yet another embodiment, the path1040is not shown, and only the discrete locations1042of the events920are displayed on the map1030.

As the current time marker1026is found on the timeline1020at the time of event two940, a location marker1044is placed on the path1040at the location of event two940. This location happens to be a scenic overlook off of Interstate 90 looking over the Badlands in South Dakota. To change the event920being viewed, a user is allowed to drag the time market1026along the timeline1020. In one embodiment, the marker1026will only “stick” on the timeline1020at time periods that define a particular event920. In this embodiment, movement of the time marker1026will cause a corresponding movement of the location marker1044. Hence, if the time marker is moved to later in the day on Jul. 4, 2013 corresponding to event three950, the location marker1044on the map interface1030will correspondingly move to Wall, S. Dak. (as can be seen in interface1100shown inFIG. 11). The interface1000can be designed so that the user can similarly move the location marker1044along the path1040between events920and have the time marker1026move correspondingly along the timeline1020. Note that while the hash markers1042identify events920on the path, there are no corresponding hash markers on the timeline1020. Of course, such marks could be added to the timeline1020. To the extent an event920exists over a long time period (such as two days), the mark1026on the timeline1020could be similarly long.

In the interface1000shown inFIG. 10, event two940was selected. The interface1000shows the content associated with this event940in content location1050. In this case, event two940was an implicit event, which means that the user of the mobile device100did not explicitly direct the device100to take any video or still photos, to record any audio, or even make any note of the fact the car stopped at this overlook. However, process600detected a trigger event (such as the car stopping for more than ten minutes) and recorded the implicit content138that was the basis for this event940. The implicit content included the fact that the car stopped at this overlook from 11:00 to 11:15 am, and it included the geographic location of this stop (within the Badlands of South Dakota). Furthermore, the user's monitoring mode in process600caused the device100to record any conversation that may have occurred. In this case, the device100captured daughter Sue exclaiming “Look at that,” followed by the mother explaining how the Badlands were formed. The audio recording1060is provided in the content area1050of interface1000, which allows the family to relive a memory that they did not explicit record.

Because the server180knows the time and location of this event940, the server180is able to augment this data with content from third party information provider servers194. In one example, the server180inquires about whether there are any locations of interest near this location. Various third parties provide public servers capable of providing this information, including Google Inc. (Menlo Park, Calif.). As this presentation grouping900is concerning a family vacation, the server180will be paying particular attention to tourist destination information. In this case, the third party information provider server194would indicate that this location is in the Badlands. As a result, the server180can populate the content area1050of the interface1000with stock photography1070showing the Badlands. In addition, the server180may include a portion of the WIKIPEDIA article on the Badlands by requesting this information from the Wikipedia server194(Wikipedia is provide by the Wikimedia Foundation, Inc. of St. Petersburg, Fla.). The server180also knows the time (11:00 am) for this event940, so it can inquire about the weather at this time and place, and add this information1074to content area1050.

Finally, the server180has within the user content data189information from many other users that have used the system180. By accessing this information, the server180may be able to identify a photograph1076taken by another, unrelated user of the server180at that very same scenic overlook and include this photograph1076in the presentation1000of event940. The use of content from other user's can be limited to that content that has been donated for public use by other users of the server system180.

Interface1100inFIG. 11displays content1150associated with event950. As was the case with interface1000, interface1100identifies at1110the time and place of this event950, displays the timeline marker1122at the appropriate location on the timeline1120for the event950, and further displays the location marker1142on the appropriate location along the path1140. For event950, the user did record explicit media content136, namely photo one1160, audio commentary taken during the event950describing photo one1160, photo two1164, and video clip1166. This content is displayed in the content presentation area1150of interface1100.

This area1150also includes some audio commentary1170that was added to this event950after-the-fact. Users are allowed to add content to events950from a variety of sources in order to enhance the presentation1100of those events. This can take the form of audio commentary1170, which helps “tell the story” of the event950. The commentary1170can be recorded the next day and connected to the event using the media organization app134on the mobile device100, or weeks later when viewing the event950through interface1100via a web browser.

Like interface1000, interface1100also includes augmentation from external data sources acquired over the Internet170. In this case, the server added weather information1180for Wall, S. Dak. on Jul. 4, 2013 at 1:00 pm, a news report1182from the local paper for that day, and stock photography of Wall, S. Dak.1184. By searching for nearby locations of interest, the server180would have identified Wall Drug and may have elected to include a Wikipedia article1186on the drug store. All of this information1180-1186relates directly to the location identified for this event950. Information related primarily to the identified time for the event950but not the location may also be included. For instance, the headline from the New York Times1188could be included for this date, which would note any major event that may have been occurring elsewhere in the world, while interesting trends on the Internet or on social media1190could also be added.

While the content1160-1190in content presentation area1150may appear cluttered inFIG. 11, the server1180allows a user to pick and choose among the content1160-1190and to format this content1160-1190as they desire. This formatting information will be stored with the event950, so that later viewers of the event950will see the preferred format for interface1100.

FIG. 12shows an interface1200for an event from a different presentation grouping. In this case, the event1320is from a “Farewell Evening with Jim” presentation grouping1300(seeFIG. 13). This event1320took place from 5:15 to 6:45 on Aug. 15, 2013, as indicated by marker1222shown on timeline1220. This event occurred on the corner of Main Street and 1st Avenue (Jimmy's Steak House), as shown by the marker1234on the path1232added to the map1230as indicated in the description1210.

The only explicit content136taken during this event1320was a group photo1250of the four friends at this restaurant. In addition, the mobile device100was allowed to record periodic conversation snippets, with at least one snippet1260being interesting enough to place on interface1200. The server180searched for locations of interest at this location1234and identified the restaurant. The server180was able to retrieve a restaurant menu1270and a photograph of the dining room from the restaurant's website1272and use this to augment the content area1240in interface1200.

This interface1200differs from interfaces1000,1100by the inclusion of a viewer-directed button1280. While the user of the mobile device100and the identified user from user data188is allowed to edit presentation groupings and events, and to format their presentations in the interfaces1000,1100,1200, the primary purpose of these interfaces1000,1100,1200is to allow other people to view this content. These viewers are not allowed to change the content in an event or change the formatting of an interface. However, through the use of a viewer content addition button1280, these viewers can add content to an event. In this case, viewers added photograph1290, and provided audio notes1292and1294.

As shown inFIG. 13, the presentation grouping1300included event one1320along with two other events1322and1324. Each event1320,1322,1324is associated with explicit content136and implicit content138and various metadata derived from this content140and other items such as calendar data212or data from event databases180,190. As shown inFIG. 13, the server180can also allow viewers to add viewer augmentation1330,1332to these events1320. These additions1330,1332can be stored along with the other event content140. In a preferred embodiment, the content area1240will automatically regroup content to reflect viewer added content1290-1294, although the user that created the presentation grouping1300would be allowed to change such formatting. In some cases, the viewer augmentation1330,1332is related to an event1320only in the context of this particular presentation grouping1300. For example, this same event1320could also be included in a presentation grouping around the topic of “my favorite restaurant meals.” In this context of this presentation grouping, the comments and contribution by Carol1330and Ted1332would be inappropriate.

In one embodiment, all of the events (such as events1320-1324) in a presentation grouping1300are displayed automatically, one after another. In this embodiment, it may be helpful to have audio clippings associated with an event1320to begin playing immediately when the event1320is first displayed. The interface1200may automatically move to the next event1322when all of the content in the first event1320has been displayed or played. In this embodiment, the audio clippings can be telling the story of the presentation grouping as each event1320-1324is presented. The creator of the presentation grouping1300can effectively tell the story for that presentation grouping1300through the audio clippings and commentary that have been added to the interface1200.

In this environment, audio augmentation1330,1332that is provided by a viewer of the interface1200may not be easily integrated into the story told by the creator of the presentation grouping1300. Nonetheless, this audio clipping may be of interest to other viewers that may want to hear the story from a different point of view. As a result, the interface1280will accept viewer augmentations and then allow later viewers to choose which commentary they hear as they watch the presentation of an event1320. Users may also be able to choose different sets of visual content to see as well. For instance,FIG. 12could present the story of saying goodbye to Jim from Jim's friends' perspective. Jim may later add his own audio commentary, photographs, and movies from that night that would tell the story from his perspective. The interface1200could group this content into two perspectives, letting viewers choose which experience they wish to have of the event1320.

In addition to manually allowing viewers to add additional content1290-1294to events, it is also possible to allow two users of the server180to merge their content together. For instance, if user one created content at an event, such as a Bruce Springsteen Concert, that user could create a presentation grouping1400that included that event1410. As part of that event1410, the user created content A1412, B1414, C1416, and D1418. This content1412-1418is stored as part of this event1410, in the user contents189on server180. A second user may have also attended that same concert and created their own presentation grouping1420containing their event1430for that concert. The second user created content E1432, F1434, and G1436for this event1430. The server would recognize that both events1410,1430occurred at the same time in the same (or close to the same) location. Furthermore, it is likely that both events1410,1430would have been associated with the concert through steps725-740of the content clustering method700.

When the first user elects to display their event1410(or format the event for later display to third parties), the server180may recognize that the user content189of the server180already contains content1432-1436from the same event that was submitted by the second user. Assuming that it has permission from the second user, some of the content1432-1436submitted by the second user could be used in presentation1440of the first user's event1410. In particular, content E1432was included without change. Content F1434may have been kept private by the second user, and therefore would not be made available to anyone else. Content G1436, however, was merged by the system with the first user's content C1416to form merged content C+G1450. This merged content might be a single video file1450containing video from content C1416and content G1436. Alternatively, the merged content might be a panoramic picture merging two separate photographs. Of course, content from numerous multiple users at the same event could be made available for other users, allowing a user to select some of the best content from others to include in their presentation1440. This might also allow dozens of separate pictures to be merged into a single panorama of the concert. The server180could even distinguish between content based on the content's precise location. At the Springsteen concert, for instance, the server180could suggest to the first user only content from users in the same approximate location within the stadium as that first user, so that the suggested photos and videos would be similar to the first user's actual view of the concert.

In addition, the first and second users may have actually attended the concert together. In this case, it may be that these users desire to merge all of their content together into a single presentation1440of the event. A single formatting would apply to the combined event, allowing both users to jointly create the presentation of the content for that event. Alternatively, as explained above, the presentation can include two perspectives of the event that a viewer can later select, with the content and commentary of the first user creating a first perspective and the content and commentary of the second user creating a second perspective.

Presentation Method

A method1500for presenting content from a presentation grouping is shown inFIG. 15. This method starts at step1505by creating a timeline. To do this, the method1500examines the event times for the events in the presentation grouping. The timeline should extend from the first event chronologically to the last, although the entire timeline need not be displayed on the user interface. At step1510, the method creates a map based on the location information found in the events in the presentation grouping. This map may utilize third-party mapping software to place the event locations on the third-party map. As explained above, the server180can create a path on the map by linking the event locations together. The server180can intelligently create the path by noting important physical features (such as roads or rivers) that link the locations together.

In one embodiment content is displayed one event at a time. At step1515, the timeline and map both display an indication of where the selected event occurs on the map, timeline or both. Note that if a user wished to include content from multiple events at the same time, the user could simply merge the two events into a single event. Alternatively, the server180could maintain the two events as separate events, but simply allow the user to display content from both events together. In this case, the timeline and map could display information for both events. Alternatively, the timeline and map could be removed from the display.

At step1520, the user formats the content from the event for presentation. In addition to organizing the layout of content, the formatting could include special effects, distortions, animations, or color filters that could be added to particular items of content. In addition, the user could specify music for the event or the entire presentation grouping. These formatting changes can be saved with the event so that all viewers would see the same formatting. In other circumstances, formatting options would be available for viewers so that they can alter the presentation of the content. These changes would not be made available to other viewers, but could be saved along with an identifier for that viewer so that the same viewer could save their formatting preferences.

At step1525, the timeline, map, and formatted content for an event are displayed to the viewer. The viewer can then select at step1530a location on the timeline or map to select a different event. At step1535, the display is updated to the selected event by updating the map and timeline, as well as the event content.

At step1540, the server180uses time and location information for the event to gather augmentation content from external servers194. This augmentation is displayed with the original content in step1545. At step1550, the same time and location information allows the server180to look for related content in its own data189from events created by other users. Assuming that these other users have granted permission to share this content, any discovered content is shared at step1555. At step1560, the server180allows viewers to submit their own augmentations, such as photographs or commentaries that they wish to add to the event presentation. At1565, these viewer-submitted augmentations are displayed for all viewers to see. Finally, at step1570, the server180may determine that two users are willing to merge content from the own events with the other user. If so, the content is merged so that the two users can collectively define and format a single event.

Steps1540-1570can take place at different times in method1500. For instances, these steps1540-1570could take place before step1505, so that the timeline and map would reflect all the augmented data that was added to the event. Alternatively, some or all of these steps could occur before step1520, which would allow the user to include the augmentation content in their desired format for the presentations grouping. In addition, none of steps in method1500should be considered mandatory, as various steps could be skipped without fundamentally changing the method1500.

Finally,FIG. 16shows media organization server180with more detail provided related to the storage of user content189. InFIG. 16, user content189is stored in a user content database1600. The database1600is stored in the memory of the media organization server180as structured data (such as separate tables in a relational database, or as database objects in an object-oriented database environment). The database programming stored on the memory of the media organization server180directs the processor182to access, manipulate, update, and report on the data in the database1600.FIG. 16shows the database1600with tables or objects for content items1610, content clusters1620, user confirmed events1630, presentation groupings1640, external augmentations1650, friend or viewer augmentations1660, and users1670. Relationships between the database entities are represented inFIG. 16using crow's foot notation. For example,FIG. 16shows that presentation groupings1640are associated with a plurality of user-confirmed events1630, which are themselves associated with one or more content clusters1620that contain one or more content items1610. Associations or relationships between the database entities shown inFIG. 16can be implemented through a variety of known database techniques, such as through the use of foreign key fields and associative tables in a relational database model. InFIG. 16, associations are shown directly between two database entities, but entities can also be associated through a third database entity. For example, a user database entity1610is directly associated with one or more content items1610, and through that relationship the user entity1670is also associated with user confirmed events1630and presentation groupings1640.

The many features and advantages of the invention are apparent from the above description. Numerous modifications and variations will readily occur to those skilled in the art. Since such modifications are possible, the invention is not to be limited to the exact construction and operation illustrated and described. Rather, the present invention should be limited only by the following claims.