Relevance Judgment
 Highly Relevant
General Rules
The video 100% answers the query
e.g. The video is categorically among the best video results for the query.
There are no videos that are significantly better than this one for answering the user’s intent.
The video does not offer anything additional that would broaden / narrow the query in a way that detracts from the overall user intent.
The video is a “best in class” answer to the user’s query.
Specific Rules
If the query is for a entity (e.g. a company or organization) the video can either be ABOUT that entity or a video FROM that entity's channel - both will qualify as Highly Relevant
 Relevant
The video matches the intended subject however it contains additional irrelevant content that detracts from the user intent
A portion of the video, but not the entire video, matches the intended subject
Query is asking for the openning scene of a movie and the result is entire movie
The video is a partial/incomplete version of the intended subject
Query is asking for a full movie and the video is showing part 2 of 3 of the movie
The Video matches and fully satisfies a minor interpretation of the query
Query is asking for "Apple" and the video is discussing the fruit rather than the company
 Related
The video matches and fully satisfies a rare interpretation of the query (such results can at best, be treated as related)
e.g., Query is looking for “Amazon” and result page is about Amazon warrior women.
The video does not match the query’s intended intent but contains related content that may be of interest and use to the user.
e.g. Query is looking for "how to change oil and oil filter" and result is "how to change oil pan gasket"
Query asks for specific content from a specific source and/or date and result is the correct content but from the wrong source and/or date
e.g. Query is "cartoons on youtube" and the result is "cartoons" on dailymotion
Correct subject, but some qualifiers in the query are not matched (such as date, video type etc.)
e.g. Query asks for a concert at a specific location/date and the result matches the location but day is wrong by 1-2 days
Satire or parody of intended subject when satire is not obviously desired by the user
e.g. Query asks for a celebrity and the result is meme of that celebrity
The video is a slideshow with audio of the intended subject
e.g. Query is "winter" and the result is a slide show of winter scenes
The video is a static image with audio of the intended subject
e.g. Query is "cat" and the result is a static image of a cat with audio of a cat meowing
 Not Related
The content of the video is unrelated to the query intent
The result is a spam video
e.g. the video says it is a movie, but rather than showing the correct content the uploader tries to get you to visit some other website to view the content instead
 Detrimental
This label indicates illegal or adult video content. A video should be rated Detrimental is any of the following apply:
X-rated content: Full frontal nudity, genitals, women’s nipples, sexual intercourse, pornography, etc.
Real footage of extreme graphic violence unsuitable for broadcast news. (eg. a beheading)
Child Porn.
NOTE: IF a query has adult intent the video should be rated on relevance. Detrimental should only be used when the query does not have adult intent but the video contains adult content.
 Cannot Access Video
The the video does not load, either on the page or by clicking the url
The page contains image(s) thumbnails of video players, but no actual video player
The page does not contain a video
The page contains an audio file without video
It is possible that some videos of Live Events don’t play after the event is over. In such a case, use the other indicators like video title, source (YT # of subscribers and other sources –whether they are genuine sites) to decide whether the live video was accessible at the time of search. If yes, and the video was relevant to the live event/topic then label the video as “Highly Relevant” instead of “Video Didn’t play”.
 Can't Judge
The video is in a foreign language and you cannot provide a well reasoned judgement.
The query is in a language you don’t understand, and the translation of the query doesn’t make sense.
Do not mark English queries as foreign.
 Language Judgment
 Does not degrade
Select this option if the language of the video is in the same language as the query.
The query is in English and the video has English audio/text
The query is for a foreign language film, but the video has appropriate subtitles that make the video understood by someone who speaks the language of the hitapp
e.g. Query is "Amelie film" (a french language film) and the video has audio in French but subtitles in English
other examples including but not limited to
e.g. Query is for foreign language song, singer, or foreign language tv show, foreign language news site, foreign youtube channel
There is no audio, or audio is not needed to understand the video
e.g. Query is "Primitive Technology" (a channel that uploads videos with no spoken dialogue or subtitles, but neither of these things are needed to understand the content) and the result is from this channel
e.g. Query is "Mimes" and result is of a mime doing their routine
e.g. Query is for sports match recap and result has foreign language commentary
 Severely degrades
The language of the video is neither the language of the query or a language communly used in the region. In these cases, the Relevance rating will always be 'Cannot Judge'
 Can't Judge
You are unable to determine the language of the video and/or do not know if the language of the video is commonly spoken in the region.
This will automatically be selected if the relevance is Detrimental, Cannot Access Video, or Can't Judge
2 hits completed in the past 0 hours and 0 minutes (0.0 hits/hour)
The page displayed should contain the video response to the query. If the page does not display below, click the link to open the page in a new browser tab.
Research the video search query below using the research icons and answer the following questions
Query: bbc 3
Query Region: United States
      
Select a rating from below that best describes the video's relevance to the query:
 [1] Highly Relevant
 [2] Relevant
 [3] Related
 [4] Not Related
 [5] Detrimental  [6] Cannot Access Video  [7] Can't Judge

Does the language of the video degrade the usefulness of the results?
 [1] Does not degrade
 [3] Severely degrades  [4] Can't Judge
https://www.bbc.co.uk/programmes/p0c3yhyh

0-7 - check correspondent controls b - Bing search g - Google search i - Bing videos search o - Google videos search t - Google translate w - Wikipedia search y - Youtube search