Patent Publication Number: US-10311038-B2

Title: Methods, computer program, computer program product and indexing systems for indexing or updating index

Description:
CROSS REFERENCE TO RELATED APPLICATION(S) 
     This application is a 35 U.S.C. § 371 National Phase Entry Application from PCT/SE2014/050027, filed Jan. 13, 2014, designating the United States, which claims priority to U.S. Patent Application No. 61/871,460, filed Aug. 29, 2013. The above identified applications are incorporated by reference. 
    
    
     TECHNICAL FIELD 
     Disclosed herein are, for example, methods, computer program, computer program product and indexing systems for indexing or updating an index. In one aspect, the disclosure is related to, inter alia, indexing time-based content (e.g., video content, audio content, and audiovisual content) to enable a user to easily find content of interest to the user. The disclosure relates to several embodiments of such a system. 
     BACKGROUND 
     The World Wide Web (WWW) grows larger and larger every day, and users access it more often on a daily basis using a variety of devices, such as Personal computers (PC), phones, tablets, cameras and Internet Protocol Television (IP-TV) devices. Mobile technologies make it easier to capture audiovisual content and different social network and video sharing sites make it possible to share such content on the Internet. As technology becomes smarter the sharing of audiovisual content becomes more common. The amount of content is growing rapidly and researchers are working on smarter ways to search, retrieve and share such content. 
     Today audiovisual content can be searched on web sites, such as YouTube (www.youtube.com, retrieved on Aug. 16, 2013), so-called “play channels” owned by traditional TV broadcasters, Video-on-Demand sites, etc. Videos are stored and tagged using conventional audiovisual metadata like title, description, duration, keywords, video category (like “Action”, “Comedy” etc.). 
     There are a number of commercial search engines available today that use proprietary algorithms for indexing and search. A first example is YouTube (www.youtube.com, retrieved on 16 Aug. 2013). Searching for a specific keyword via YouTube can provide three different types of results: 1) Videos (e.g., a link to a video which is relevant to the search query); 2) Playlists (e.g., link to a playlist, an ordered set of videos, which are relevant to the search query); and 3) Channels (e.g., a link to a channel (user account) which contains videos relevant to the search). 
     Another example is Vimeo (www.vimeo.com, retrieved on 16 Aug. 2013), wherein the search function is segmented into different types of content, and the user is required to select which type of search results that is expected. By default, a search will display a list of matching videos, but the user is able to select to instead search for people, channels, groups or forum. 
     Yet another example is Google (www.google.com, retrieved on 16 Aug. 2013). Here a standard web search can present results ordered hierarchically. The most common result on a web search is a list of web pages which match the search query. For each of these, the user can expand to see exactly where in the page the search query is matched. Additionally, relevant pages within the same site are displayed adjacently, with the most relevant result for said site being displayed on top. The Google search engine searches web pages rather than video content. 
     What is desired is, for example, a search engine system having an improved content indexing system that enables a user to more easily find the content the user is searching for. 
     SUMMARY 
     Described herein are various embodiments for indexing or updating an index. An object of at least one embodiment may be to enable improved search accuracy for multimedia content in relation to the prior art mentioned in the background. Another object of at least one embodiment may be to enable more effective techniques for generating accurate search indices for multimedia content. Embodiments of the indexing and searching system disclosed herein use a time range, hereinafter sometimes called Window of Relevance (WoR), associated with keywords (which are also referred to as “tags”). The WoR enables a more effective display of search results in, for example, a graphical user interface (GUI) displayed by a user equipment (UE) (e.g., a computer, smartphone, etc), by displaying at least one word adjacent to a displayed keyword used in a search. The at least one word makes it easier for the user to guess or know if a hit in the search result is relevant or not. Other possible technical advantages is that the implementation of some embodiments of the indexing and search system becomes more dynamic and/or generic by e.g. making the indexing more unrelated to more “stationary segments” and their size. Some embodiments also make the searching multimedia content faster than conventional methods. 
     In one aspect there is provided a method performed by an indexing system for indexing or updating an index for content. In some embodiments, the method comprises for each one of a plurality of keywords in a data storage system, wherein each keyword is associated with a modality and at least a first time stamp, retrieving from the data storage system a keyword and its associated first time stamp. The method also comprises defining a time range for the keyword and determining which other keywords in the data storage system are associated with a time stamp within the time range. The method further comprises updating the index with the other keywords that were determined to be associated with a time stamp within the time range. The method may also comprise updating the index with a rank value and modality of each one of the determined other keywords with time stamps within the time range. 
     In some embodiments, the method also comprises, before the step of retrieving the keyword from the data storage system, on at least a plurality of frames of a video: identifying one or more features present in the frames, wherein each one of the features are of at least one modality. The method according to the embodiments also includes identifying, for each identified feature, a keyword associated with the identified feature and generating the at least one time stamp for each one of the identified keywords. The method according to the embodiments further includes storing the keywords and their respective time stamps in the data storage system. 
     In some embodiments, the time range is defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp. 
     In some embodiments, the keyword is further associated with a second time stamp and the time range starts a certain time before the time of the first time stamp and ends a certain time after the second time stamp. 
     In another aspect there is provided an indexing system. The indexing system comprises a computer system and data storage system. The data storage system contains instructions executable by the computer system. The indexing system is operative to, for each one of a plurality of keywords in the data storage system, wherein each keyword is associated with a modality and at least a first time stamp: retrieve from the data storage system a keyword and its associated first time stamp and define a time range for the keyword. The indexing system is further operative to determine which other keywords in the data storage system are associated with a time stamp within the time range. The indexing system is further operative to update an index with the other keywords that were determined to be associated with a time stamp within the time range. 
     In another aspect there is provided a computer program for indexing content. In some embodiments, the computer program comprises computer readable instructions. The computer readable instructions are configured such that when run on a computer system they cause the computer system to retrieve a keyword from a plurality of keywords stored in a data storage system, wherein the retrieved keyword is associated with a modality and at least a first time stamp, and define a time range for the keyword. The computer readable instructions further cause the computer system to determine which other keywords in the data storage system are associated with a time stamp within the time range. The computer readable instructions further cause the computer system to update an index with the other keywords that were determined to be associated with a time stamp within the time range. 
     In another aspect there is provided an indexing system for indexing or updating an index for content. In some embodiments, the indexing system comprises means for retrieving from a data storage system a keyword and a time stamp associated with the keyword. The indexing system further comprises means for defining a time range for the keyword. The indexing system further comprises means for determining which other keywords in the data storage system are associated with a time stamp within the time range. The indexing system further comprises means for updating an index with the other keywords that were determined to be associated with a time stamp within the time range. 
     In another aspect there is provided a content indexing method performed by a indexing and search system. In some embodiments, the method comprises storing in a data storage system a plurality of keywords, the plurality of keywords including a first keyword and a second keyword. The method further comprises defining a time range for the first keyword. The method further comprises, after defining the time range for the first keyword, determining that the second keyword falls within the defined time range. The method further comprises, in response to determining that the second keyword falls within the defined time range, updating a first set of keywords by adding the second keyword to the first set of keywords, wherein the first set of keywords is associated with the first keyword. 
     The above and other aspects and embodiments are described below with reference to the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments. 
         FIG. 1  illustrates functional components of an indexing and search system according to some embodiments. 
         FIG. 2  illustrates an example format for an index according to some embodiments. 
         FIG. 3  is a flow chart illustrating a process, according to some embodiments, that may be performed by a query processor. 
         FIG. 4  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 5  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 6  illustrates an example format for an index according to some embodiments. 
         FIG. 7  is a flow chart illustrating a process, according to some embodiments, that may be performed by a query processor. 
         FIG. 8  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 9  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 10  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 11  is a flow chart illustrating a process, according to some embodiments, that may be performed by an indexing system. 
         FIG. 12  is a block diagram of an indexing system according to some embodiments. 
         FIG. 13  is a diagram illustrating components of an indexing system according to some embodiments. 
         FIG. 14  is a block diagram of a query processor according to some embodiments. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  illustrates functional components of an indexing and search system  100  according to some embodiments. Indexing and search system  100  includes an indexing system  104  for creating an index  106  based on the contents of a content library  102  (e.g., a set of one or more media files, such as video files containing audiovisual data). For example, index  106  may include a set of records (a.k.a., “vectors”) where each record includes: a keyword generated from content within content library  102  and information for locating the content (e.g., a filename or other identifier and a time stamp). Indexing and search system  100  further includes a query processor  108  for: receiving a query  111  input by users of system  100 , processing the query by using keyword(s) included in the query and index  106  to find content that matches the user&#39;s query, and providing a search result  112  to the user. In some embodiments, indexing and search system  100  is a distributed computer system where at least some of the functions of each functional component are performed by a different computer and the different computers may be remotely located from each other. In other embodiments, indexing and search system  100  is a monolithic computer system where each of the functions of each functional component is performed by one computer. In other embodiments, indexing and search system  100  may be a combination of such computer systems. 
       FIG. 2  illustrates an example format for index  106  according to some embodiments. As illustrated in  FIG. 2 , index  106  is an index created for a video library containing one or more videos. As further illustrated in  FIG. 2 , index  106  may include a set of records  202 , where each record includes a keyword  204  and video identifier (video id)  216  identifying a video with which the keyword is associated. Each record  202  may also include other information, such as: a modality identifier  206  identifying a modality with which the keyword is associated (e.g., text, image, audio); a start time stamp (“start TS”)  208  identifying a first position (e.g., first frame) within the video; an end time stamp (“end TS”)  210  identifying a second position (e.g., second frame) within the video; a rank value  212  identifying a rank for the keyword; and a set  214  of other keywords that overlap with the keyword. For example, record  202   a  includes a set of values, including among other values: a keyword  204   a , a start TS  208   a , an end TS  210   a , and a set of keywords  214   a  that overlap keyword  204   a ; and record  202   b  includes a set of values, including among other values: a keyword  204   b , a start TS  208   b , an end TS  210   b , and a set of keywords  214   b  that overlap keyword  204   b . More specifically, as  FIG. 2  shows, an image of president Obama and an image of a flag appeared in video  1  from position  9  (e.g., frame  9 ) of video to position  12  (e.g., frame  12 ) of video  1 . 
     Even though it is not illustrated in index  106 , the index  106  may also comprise the ranking/weight value of the respective matching keywords in the list  214 . 
     In some embodiments, the rank assigned to a keyword k depends on the modality of k and whether k overlaps with other keywords. For example, if we assume k overlaps with N other keywords, then we can use a formula of the following form to calculate the rank R_k to assign to keyword k: R_k=Mv_k+Ov_k,1+Ov_k,2+ . . . +Ov_k,N, where Mv_k is a value assigned to the modality of k and Ov_k,i is an overlap value that is dependent on the degree of overlap between the keyword k and keyword i. For instance, Ov_k,i may equal: ((k_end−i_start)/(i_end−i_start))*Mv_i, where k_end is the end TS value for keyword k, i_start is the start TS for keyword i, i_end is the end TS for keyword i, and Mv_i is the modality value for keyword i. Using the above formula and assuming that the modality value for keywords having a “text” modality is 5, we can calculate the rank for the keyword “intro” as follows: R_intro=5+((8−6)/10−6))*5=7.5 
     Referring now to  FIG. 3 ,  FIG. 3  is a flow chart illustrating a process  300 , according to some embodiments, of using index  106  to produce a search result in response to a query from a communication device  101  (see  FIG. 1 ), which query could have been initiated by a user via a user interface of the communication device  101 . Process  300  may be performed by query processor  108 . 
     In step  302 , query processor  108  receives a query that was input by the user into communication device and transmitted by the communication device  101  to query processor  108 , where the query comprises a keyword (e.g., Obama). In some embodiments, query processor  108  includes a Hypertext Transfer Protocol (HTTP) server. Thus, in such embodiments, the user may input the basis (such as the keyword) for the query into a web page form and initiate the submission of the query to the query processor  108  in an HTTP GET message. That is, for example, the web page form may have a text input box to enable the user to input a text based query and a “search” button. When the user activates the search button after inputting into the text input field a query, the user&#39;s browser causes the communication device  101  on which the browser is running to transmit to query processor  108  an HTTP GET message that includes the text (i.e., query) that the user input into the text input box. 
     In step  304 , query processor  108  determines that the keyword included in the query matches a first keyword (e.g., keyword  204   a  “Obama”) included in the index  106 . 
     In step  306 , query processor  108  obtains a second keyword (e.g., keyword  204   b  “flag”) that overlaps with the first keyword  204   a  (e.g., Obama). For example, query processor may obtain the second keyword  204   b  by retrieving from index  106  record  202   a , which includes a set of keywords  214   a  (i.e., CNN and flag) that overlaps with the Obama keyword  204   a.    
     In step  308 , query processor  108  generates search result information, wherein the search result information comprises: a) a first locator identifying a first image file associated with the first keyword (e.g., a thumbnail image of President Obama) and a second locator identifying a second image file associated with the second keyword (e.g., a thumbnail image of a flag); and/or b) the first keyword and the keywords that overlap the first keyword (e.g., the second keyword). Query processor  108  may obtain the first and second locators from information contained in index  106 . For example, while not shown, the first locator may be stored in a field of record  202   a  and the second locator may be stored in a field of record  202   b . As another example, query processor  108  may construct the first and second locators using information contained in records  202   a  and  202   b , respectively. 
     In step  310 , query processor  108  adds the generated search result information to a search results page. 
     In step  312 , query processor  108  provides the search results page to the communication device  101  which displays the page to the user. For example, the search results page may be in the form of a Hypertext Markup Language (HTML) document) and query processor may use HTTP to transmit the HTML document to the user&#39;s communication device  101  (e.g., laptop computer, smartphone, etc). 
     In this way, indexing and search system  100  provides the advantage of not only providing the user with information associated with the keywords contained in the user&#39;s query (e.g., Obama), but also providing to the user additional information associated with keywords (e.g., flag) that overlap the keywords of the user&#39;s query. This additional information can be very advantageous to the user because it provides additional contextual information that may enable the user to find the search result from the search results page that bests matches what the user was searching for. 
     Referring now to  FIG. 4 ,  FIG. 4  is a flow chart illustrating a set of steps  400  (a.k.a., process  400 ), according to some embodiments, that may be performed by indexing system  104  to create index  106 . Process  400  may begin with step  402 , where indexing system  104  obtains (e.g., retrieves) a video from library  102 . 
     In step  404 , indexing system  104 , for each video frame of the video, analyzes the frame to produce frame data, which frame data includes a set of keywords. For example, in step  404 , indexing system  104  may use image recognition software to recognize and label images found within each video frame. The label given to an image identified in a video frame becomes a keyword for that video frame. In this way, indexing system  104  generates keywords for a video. 
     In step  406 , for each keyword obtained from performing step  404 , indexing system  104  analyzes the frame data to determine for the keyword a time range (a.k.a., WoR) (e.g., a video frame range), which consists of a starting time stamp and an ending time stamp. Time may be measured in units of seconds, units of frames, or any other unit that can be used to identify a position (e.g., frame) within the video. Thus, for example, in some embodiments, a time stamp is a value having units of time (e.g., seconds), while in some other embodiments, a time stamp is a value having units of frames.  FIG. 5 , discussed below, illustrates a process for determining a time range for each of the keywords. 
     In step  440 , which may be performed concurrently with one or more of steps  404 - 406 , indexing system  104  analyzes the audio portion of the video to extract keywords from the audio portion. In step  445 , indexing system  104  assign a time range to each keyword. After step  445 , process  400  proceeds to step  408 . 
     In step  408 , for each keyword obtained from performing steps  404  and  440 , indexing system  104  determines other keywords that “overlap” the keyword. In some embodiments, a first keyword overlaps a second keyword if there is any overlap in the time ranges for the two keywords. 
     In step  410 , for each keyword obtained from performing steps  404  and  440 , indexing system  104  updates the index  106  with the keyword and the determined other keywords that overlap the keyword. For example, in step  410 , for each keyword, indexing system  104  inserts into index  106  a new record for the keyword, wherein the keyword is stored in the “keyword” field  204  of the record and the keywords that were determined to overlap with the keyword are stored in the “overlap” field  214  of the record. As shown in  FIG. 2 , the new record that is inserted into index  106  may also include a modality value stored in the “modality” field  206 , a rank value stored in the “rank” field  212 , and a video identifier value that identifies the video retrieved in step  402  in the “video id” field  216 . In some embodiments, the modality value is one of: image, text, and audio. Additionally, in some embodiments, the rank value for a keyword is a value that is calculated based on, for example, one or more of: the modality associated with the keyword, the length ( 1 ) of the time range associated with the keyword (e.g., 1=end time stamp−start time stamp), and the number of keywords that overlap with the keyword. 
     Referring now to  FIG. 5 ,  FIG. 5  is a flow chart illustrating a process  500 , according to some embodiments, that may be performed by indexing system  104 . Process  500  may begin with step  502 , in which indexing system  104  obtains (e.g., retrieves) a video from library  102  and sets variable named “i” equal to 1. In step  504 , indexing system  104  analyzes frame i of the video to produce a set of keywords for frame i. 
     In step  506 , for each keyword in frame i, indexing system  104  sets start-index=i and adds to a Temporary Keyword Set (TKS) a keyword vector (keyword, start-index) for the keyword. In step  508 , indexing system sets i equal to the value i+x, where x is a whole number greater than zero. In step  510 , indexing system  104  determines whether frame i exists in the video. If it does not, then process proceeds to step  514 , otherwise it proceeds to step  512 . In step  512 , indexing system  104  analyzes frame i of the video to produce a set of keywords for frame i. 
     In step  514 , for each keyword vector included in TKS that does not include a keyword that is included in the set of keywords for frame i, indexing system  104  removes from TKS the keyword vector (keyword, start-index) for the keyword, sets end-index equal to the value of i, and stores in a data storage system  1206  (see  FIG. 12 ) (e.g., a record of index  106 ) the keyword vector (keyword, start-index, end-index) for the keyword. In some embodiments, rather than store in data storage system  1206  the keyword vector (keyword, start-index, end-index) for the keyword, indexing system  104  stores in the data storage system  1206  the following keyword vector for the keyword: (keyword, start-index−offset 1 , end-index+offset 2 ), where offset 1  and offset 2  are values greater than zero. 
     In step  516 , for each keyword in the set of keywords for frame i that is not included in TKS, indexing system  104  sets start-index=i, and adds a keyword vector (keyword, start-index) for the keyword to TKS. 
     In the above manner, indexing system can determine a time range (e.g., a video frame range) for each keyword. 
     Referring now to  FIG. 6 ,  FIG. 6  illustrates an example format for index  106  according to other embodiments. The embodiment of index  106  shown in  FIG. 6  differs from that shown in  FIG. 2  in that the index shown in  FIG. 6  does not include the “overlap” field. This is because in some embodiments, query processor  108  determines overlapping keywords in real-time in response to a query, which query may include one or more offset values in addition to one or more keywords. 
     Referring now to  FIG. 7 ,  FIG. 7  is a flow chart illustrating a process  700 , according to some embodiments, that may be performed by query processor  108 . Process  700  may begin with step  702 , in which query processor  108  receives a query transmitted by the communication device  101  (e.g., a query the user input into communication device  101 ), where the query includes a keyword and an offset value (OV). 
     In step  704 , query processor  108  determines that the keyword included in the query matches a first keyword (e.g., a keyword included in index  106 ). 
     In step  706 , query processor  108  obtains a first time stamp (FTS) associated with the first keyword. For example, the FTS may be a start time stamp associated with the first keyword (e.g., the start TS value that is stored in the same record of index  106  in which the first keyword is stored). 
     In step  708 , query processor  108  defines the time range using FTS and OV. For example, the time range may be defined by a start time stamp and an end time stamp, where the start time stamp is equal to (FTS−OV) and an end time stamp is equal to a second time stamp (STS) plus OV (i.e., STS+OV). STS may be an end time stamp associated with the first keyword. 
     In step  710 , query processor  108  determines a set of keywords other than the first keyword that are located within the defined time range. That is, as described above, the keywords that overlap with the first keyword are found using, for example, the start and end time range values stored in index  106 . 
     In step  712 , query processor  108  generates a search result using the first keyword and the determined set of other keywords. In step  714 , query processor  108  provides the search result to the communication device  101 . 
     In the above manner, the overlapping keywords are determined in real-time in response to a query. 
       FIG. 8  is a flow chart illustrating a process  800  according to some embodiments, that may be performed by indexing system  104  to create the index  106  shown in  FIG. 6 . Process  800  may begin with step  802 , where indexing system  104  obtains (e.g., retrieves) a video from library  102 . 
     In step  804 , indexing system  104 , for each video frame of the video, analyzes the frame to produce frame data, which frame data includes a set of keywords. For example, in step  804 , indexing system  104  may use image recognition software to recognize and label images found within each video frame. The label given to an image identified in a video frame becomes a keyword for that video frame. 
     In step  806 , for each keyword obtained from performing step  804 , indexing system  104  analyzes the frame data to determine for the keyword at least a starting time stamp. In some embodiments, for each keyword, indexing system  104  determines a starting and an ending time stamp for the key word. 
     In step  840 , which may be performed concurrently with one or more of steps  804 - 806 , indexing system  104  analyzes the audio portion of the video to extract keywords from the audio portion. In step  845 , indexing system  104  assigns a starting time stamp to each of the extracted keywords from step  840 . After step  845 , process  800  proceeds to step  810 . 
     In step  810 , for each keyword obtained from performing steps  804  and  840 , indexing system  104  updates the index  106  with the keyword and at least the keyword&#39;s starting time stamp. For example, in step  810 , for each keyword, indexing system  104  inserts into index  106  a new record for the keyword, wherein the keyword is stored in the “keyword” field  204  of the record, the starting time stamp is stored in the “start TS” field  208 , and, if an ending time stamp exists, the ending time stamp is stored in the “end TS” field  210 . As shown in  FIG. 6 , the new record that is inserted into index  106  may also include a modality value stored in the “modality” field  206 , a rank value stored in the “rank” field  212 , and a video identifier value that identifies the video retrieved in step  802  in the “video id” field  216 . In some embodiments, the modality value is one of: image, text, and audio. Additionally, in some embodiments, the rank value for a keyword is a value that is calculated based on, for example, the modality associated with the keyword and/or the length ( 1 ) of the time range associated with the keyword (e.g., 1=ending time stamp−starting time stamp). 
     As explained upon above, a feature of various embodiments is an algorithm that extracts and indexes keywords within a window of relevance (WoR). The WoR is in one embodiment set for each key word, t(keyword)±t(WoR), wherein t(keyword) is a time stamp associated with a tag/keyword in e.g. a multimedia content, such as a video, and t(WoR) is a time, which may be fixed for all modalities, fixed but different for different modalities, such as one t(WoR) value for features/faces in images/frames in a video and another value for derived keywords from speech and still another value for derived keywords from sound and yet another value for text (such as derived words from subtitles). Other ways of setting the WoR than fixed values in the above mentioned way, may be to set it more dynamic so that at least one other keyword comes within the window of relevance, so that it is easily enabled that at least one keyword is displayed in a search result list adjacent to the main keyword, which in its turn may be highlighted to make it easier for the user to judge the relevance of the hit in the result list or result view. 
     The WoR may in some embodiments also be made to be dependent on the type of keyword itself or a general tag associated with the content. An action movie might for example be made to have different window of relevance than e.g. a romantic, slow paced movie, whereas talk shows may have still another set of values for the WoR. Yet other ways of setting the WoR is to make it dependent on whether the keyword is present in more than one modality at the same time, e.g. has a rank value above a certain threshold, and/or dependent on the length of the video. 
     In some embodiments, the algorithm included into the indexing process enables an improvement of the search capabilities for audio visual content. The indexing and window of relevance algorithm here, in some embodiments, takes a given number of frames per second in a video and processes each frame as an image. The images are then analyzed for metadata like text (using e.g. OCR), faces (using face detection algorithms), voice, objects and other metadata and could in some embodiments be combined with manual tagging of the content. All metadata will be indexed, see  FIG. 9 , which comprises some steps performed at least partly by the algorithm. 
     As illustrated in  FIG. 9 , a method according to an embodiment may include the following features: 
     I. A multimedia content, in this example a video, is captured or uploaded to a server, e.g. the ISS. 
     II. The WoR algorithm start analyzing the uploaded video, e.g. already when the uploading is made, and in the following way: 
     a) For each modality (image, voice, text, sound etc) analyze each frame in the video: 
     i) For each frame:
         1. Extract keywords, and for each one of the keywords an associated modality as well as at least one time stamp, t_start and optionally also a second associated time stamp t_end. In one embodiment there may, for example, be a first time stamp and a second time stamp associated with a person whose face is included in a video stream for during a certain time of the video. The two values may typically be when the person first “enters the screen” and the second value when the person is not visible any more. Spoken words derived from speech or subtitles may on the other hand be associated with just the first time stamp. Background sound, such as a song or soundtrack typically is rather long and may therefore with advantage be associated with a first and a second time stamp.   2. Save data to Vector Space, i.e typically a Random-access memory used as intermediate memory before data is stored in the database. For example, a json-file (.json) including keywords, related times and modality is created and uploaded to the Vector Space. Here the data may be sorted according to the first time stamp.       

     ii) For each keyword in Vector Space/RAM:
         1) Rank keyword (modality): e.g. rank keyword according to the associated modality.   2) Define WoR, i.e. a time range for the keyword   3) Add/associate/link related keywords within WoR to the keyword   4) Add modality: add modality value/score.   5) Update database, which in one embodiment is an SQL database.   6) Update Index, wherein the index in one embodiment is comprised in the commercially available search engine software Lucene which run on the commercially available search platform Apache Solr. Every update can in this “Apache” embodiment be what in that use case normally is called a document.       

     The steps define WoR, Add related keywords and add modality builds up the meta data and the structure for the data which shall be inserted into the database. In the Add modality step, it is defined what you actually intends to store in the database, such as info indicating modality and what the rank value is for that modality, but in some embodiments it may be sufficient with only indicating the rank value if there is no need to indicate modality in the index and/or database. In case json-files are created, the json-files may be aggregated and kept in the vector space and sorted according to the first time stamp. Crosslinking can be done to provide the ranking and matching keywords. In preparation for at least one of the two next updating steps, there may in one embodiment in practice be that a new list is created and which include ranking and modality is included and wherein every row is formatted with every first time stamp. 
     In case of a video and persons tagged in the video, the algorithm may find a certain amount of people in the video and it may be checked if the person shows up in a plurality of times in the video (plurality=more than one). The ranking of the person/object in the index can then be made dependent on how long the person/object is visible in the movie and even get a higher ranking if he/she/it is mentioned or talks a lot in the movie. 
     Index  106  shown in  FIG. 2  illustrates an example of data stored in the index. Here there is a row for each keyword/tag and each row includes a field for the keyword itself or an ID for a keyword, data indicating modality (such as text, speech, image, sound), the actual start time of the WoR, the actual end time of the WoR (where time zero would be at the very beginning of the video), rank associated with the keyword (where the rank is dependent on modality and optionally also other parameters as disclosed in US Publication No. 2013/0226930), matching keywords or their ID, and the related video_id. Optionally, there may also be a column for at least the first time stamp t_start, but may also include the second time stamp t_end. 
     Hence, when a search is made for the keyword “Obama”, the index table displaying the most relevant hits in the GUI in e.g. a web browser connecting to the indexing and search system  100 , may also display e.g. adjacent to a highlighted word “Obama” or a picture of Mr President Obama taken from the video (to more readily show that Obama is actually visible in that hit and not only talked about etc) display the text “CNN”. By displaying CNN next to the picture of Mr President Obama, the searcher should be able to more readily guess the context and relevance of the video in relation to what is actually searched for. 
     Another, somewhat different embodiment is described in the following: 
     The window of relevance algorithm is here included in the search function, e.g. as an added feature to an existing query processor  108  (a.k.a., search engine). In this case the window of relevance can be set by the user via a GUI in the communication device  101 , such as a User Equipment like a tablet, mobile phone, laptop, or a communications device embedded in a vehicle i, and be modified for each search query.  FIG. 7  illustrates an example process according to this embodiment. 
     In this embodiment the indexing table/index contains all key words, modalities and time stamps according to the example below. 
     The algorithm here analyzes videos that have been or is currently being uploaded to the server. For example, if a video is currently being uploaded, the algorithm is run by the server or the UE and here looks and tries to recognize faces and text and optionally also for sound. A json-file (.json) including keywords, related times and modality is created. 
     In a first step, the communication device  101  submits a search query to the server. 
     In steps 2-3, the server run another part of the WoR algorithm on the query results 
     a) The query results consist of a ranked list of matching keywords, t_start, and modality 
     b) For each keyword in list
         i) Define WoR by: WoR=(t_start−t(WoR), t_end+t(WoR) or WoR=t_start+/−t(WoR). The WoR may as mentioned above be dependent on modality and whether t_end is used for that modality.   ii). Fetch related keywords from index table within WoR for said keyword   iii) Add related keywords to result list
           1. Associated to said keyword   2. View its modality (e.g. in order to enable decision for how the related keywords and/or the keyword should be presented)   
               

     c) Optionally keywords which are included in overlapping WoRs may be boosted in rank. 
     In step 3, return updated ranked list of search results. 
     The embodiments described herein may enable detailed and precise search of audio visual content. 
     Referring to  FIG. 10 , a method  1000  performed by a computer system, such as the indexing system  104  or the indexing and search system  100 , for indexing or updating an index for multimedia content, (and optionally also ranking and displaying search results upon a search request for multimedia content), the method comprising for each one of a plurality of keywords in a memory (a.k.a., data storage system  1206 ) (e.g. the vector space above), wherein each keyword is associated with a modality and at least one time stamp: 
     retrieving from the data storage system  1206  the keyword and the at least one time stamp (step  1002 ), 
     define a time range (WoR) for the keyword (step  1004 ), 
     determining if there are other keywords in the data storage system  1206  which are associated with a time stamp within the time range (i.e., determine which other keywords in the data storage system  1206  are associated with a time stamp within the defined time range) (step  1006 ), 
     updating index  106  with, for each one of the keywords in the data storage system  1206 , found other keywords with time stamps (matching keywords) within the time range (step  1008 ). 
     The method may also comprise updating the index and/or database with a rank value and modality of each one the other keywords with time stamps within the time range (i.e. the matching keywords). 
     Referring to  FIG. 11 , the method may comprise the following steps performed by the computer system before the step of retrieving the keyword from the data storage system  1206 , on at least a plurality of frames of a video: 
     identifying one or more features present in the frames, wherein each one of the features are of at least one modality/media type (step  1102 ); 
     identifying, for each identified feature, one or more respective keywords associated with the identified feature (step  1104 ); 
     generating the at least one time stamp for each one of the identified keywords (step  1106 ); and 
     storing the keywords and their respective relevances and time stamps in the data storage system  1206  (step  1108 ). 
     The method may further include determining, for each identified keyword associated with an identified feature, a respective relevance of the keyword, wherein the respective relevance is dependent on a weight associated with the respective modality/media type of the identified feature. 
     The time range may be defined as starting a certain time before the time of the first time stamp and ending the certain time after the first time stamp. The time range, may alternatively, e.g. for images, start a certain time before the time of the first time stamp and end the certain time after a second time stamp associated with the keyword. 
     Apparatus, computer program and computer program products corresponding to the embodiment illustrated by the method is disclosed by the above disclosure. An embodiment of an apparatus, such as a server/server device/server host, may correspondingly be said to be a server device comprising a processor and memory, said memory containing instructions executable by said processor whereby said server device is operative to, for each one of a plurality of keywords in the memory, wherein each keyword is associated with a modality and at least one time stamp: retrieve the keyword and the at least one time stamp from the memory or the second memory; define a time range (WoR) for the keyword; determine if there are other keywords in the memory which are associated with a time stamp within the time range; and update an index and/or a database with, for each one of the keywords in the memory, found other keywords with time stamps (matching keywords) within the time range. 
     The memory should above not be seen as comprising just one memory, but should also be interpreted as also comprising embodiments where two or more memories are utilized, for example when the instructions are stored in one physical memory part and the keywords and time stamps etc are stored in another memory part in the server device. 
       FIG. 12  is a block diagram of an embodiment of indexing system  104 . As shown in  FIG. 12 , indexing system  104  may include or consist of: a computer system (CS)  1202 , which may include one or more processors  1255  (e.g., a general purpose microprocessor) and/or one or more circuits, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), a logic circuit, and the like; a network interface  1203  for use in connecting indexing system  104  to a network  110  (e.g., the Internet or other network); and a data storage system  1206  (also referred to as “memory”), which may include one or more non-volatile storage devices and/or one or more volatile storage devices (e.g., random access memory (RAM)). In embodiments where indexing system  104  includes a processor  1255 , a computer program product (CPP)  1233  may be provided. CPP  1233  includes or is a computer readable medium (CRM)  1242  storing a computer program (CP)  1243  comprising computer readable instructions (CRI)  1244 . CRM  1242  is a non-transitory computer readable medium, such as, but not limited, to magnetic media (e.g., a hard disk), optical media (e.g., a DVD), solid state devices (e.g., random access memory (RAM), flash memory), and the like. In some embodiments, the CRI  1244  of computer program  1243  is configured such that when executed by computer system  1202 , the CRI  1244  causes the indexing system  104  to perform steps described above (e.g., steps described above with reference to the flow charts shown in the drawings). In other embodiments, indexing system  104  may be configured to perform steps described herein without the need for a computer program. That is, for example, computer system  1202  may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software. 
       FIG. 13  is a functional block diagram illustrating components of indexing system  104 , according to some embodiments. As illustrated in  FIG. 13 , in some embodiments, indexing system  104  includes: means  1301  for retrieving from a memory a keyword and a time stamp associated with the keyword; means  1302  for defining a time range for the keyword; means  1303  for determining which other keywords in the memory are associated with a time stamp within the time range; and means  1304  for updating an index with said other keywords that were determined to be associated with a time stamp within the time range. 
       FIG. 14  is a block diagram of an embodiment of query processor  108 . As shown in  FIG. 14 , query processor  108  may include or consist of: a computer system (CS)  1402 , which may include one or more processors  1455  (e.g., a general purpose microprocessor) and/or one or more circuits, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), a logic circuit, and the like; a network interface  1403  for use in connecting query processor  108  to a network  110  (e.g., the Internet or other network); and a data storage system  1406 , which may include one or more non-volatile storage devices and/or one or more volatile storage devices (e.g., random access memory (RAM)). In embodiments where query processor  108  includes a processor  1455 , a computer program product (CPP)  1433  may be provided. CPP  1433  includes or is a computer readable medium (CRM)  1442  storing a computer program (CP)  1443  comprising computer readable instructions (CRI)  1444 . CRM  1442  is a non-transitory computer readable medium, such as, but not limited, to magnetic media (e.g., a hard disk), optical media (e.g., a DVD), solid state devices (e.g., random access memory (RAM), flash memory), and the like. In some embodiments, the CRI  1444  of computer program  1443  is configured such that when executed by computer system  1402 , the CRI causes the query processor  108  to perform steps described above (e.g., steps described above with reference to the flow charts shown in the drawings). In other embodiments, query processor  108  may be configured to perform steps described herein without the need for a computer program. That is, for example, computer system  1402  may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software. 
     While various embodiments of the present disclosure are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context. 
     Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.