NQ数据集上会出现不相关文档的相关性分数明显高于相关文档的情况

#17
by Xvhui - opened

一个具体的测试例子如下。后两个文档相较前三个与问题更相关,但是分数最低

from FlagEmbedding import FlagReranker
reranker = FlagReranker('BAAI/bge-reranker-large', use_fp16=False)

score = reranker.compute_score([['the cast of a good day to die hard', "One Life to Live -- Park's decision. Though not one of the reasons given by Prospect Park, 'Deadline Hollywood' suggested that the company's lack of success in finding a cable network to carry the show may have been instrumental in the company's decision to not pursue the project. Despite its fruitless attempt to save the series, Prospect Park had succeeded in retaining 13 actors to sign for the online venture, compared to only two actors for 'All My Children'. Matriarch actress Erika Slezak (Victoria Lord) was among the 13. The 12 other actors were Melissa Archer (Nathalie Buchanan), Kassie DePaiva (Blair Cramer), Michael Easton (John"], 
['the cast of a good day to die hard', "Dayzee Leigh -- on life. She became the key person who encouraged Stephanie to undergo treatment, which had ultimately saved her life. The show's official website describes Dayzee as: 'Once a resident of Los Angeles' most impoverished neighborhood of Skid Row, Dayzee still serves as a guardian angel to the community she once called home.' Romantically and optimistically, Dayzee can be flirtatious and allows everyone she comes across to view the world with 'a new set of glasses'. In October 19, 2011, it was announced that Gina Rodriguez would join the show as an unprivileged friend of Dayzee's, Beverly. Beverly is the first"], 
['the cast of a good day to die hard', "Taran Killam -- the March 20, 2006 episode, 'Life Among the Gorillas' and the final one was the March 24, 2014 episode, 'The End of the Aisle'. In 2015, Killam provided the voice of the titular character on the PBS Kids show 'Nature Cat', along with 'SNL' alumni Kate McKinnon, Bobby Moynihan, and Kenan Thompson. On January 17, 2017 Killam succeeded Rory O'Malley in the role of King George III in the Broadway musical 'Hamilton'. Killam produced, scripted, directed and starred in the comic hitman film 'Killing Gunther', given a general release in fall 2017. Arnold Schwarzenegger, who played Gunther in the film,"], 
['the cast of a good day to die hard', "A Good Day to Die Hard -- A Good Day to Die Hard A Good Day to Die Hard is a 2013 American action thriller film and the fifth installment in the 'Die Hard' film series. The film was directed by John Moore and written by Skip Woods, and stars Bruce Willis as John McClane. The main plot finds McClane travelling to Russia to get his estranged son, Jack, an undercover CIA agent, out of prison. He is soon caught in the crossfire of a global terrorist plot. Alongside Willis, the film also stars Jai Courtney, Cole Hauser, Yuliya Snigir and Sebastian Koch. Talks of a fifth"], 
['the cast of a good day to die hard', "A Good Day to Die Hard -- were cut from the Extended release. Sebastian Koch played the film's primary antagonist, Yuri Komarov, while Yuliya Snigir and Cole Hauser featured as secondary characters Irina and Collins. The cast was completed by actors Amaury Nolasco as a friend of McClane, Pasha D. Lychnikoff as a taxi driver, and Megalyn Echikunwoke, Anne Vyalitsyna, and Ivan Kamaras in smaller roles. Production began in Hungary in April 2012, with the capital Budapest standing in for Moscow. A military shooting range near Hajmáskér was used for shooting live ammunition, while vehicular stunts were shot at the Hungaroring, a Formula One racing circuit in"]])
print(score)

[9.53571605682373, 8.141484260559082, 8.54487419128418, 7.349508762359619, 6.388111114501953]

另外,把文档加上lower()全部变成小写后,分数变化为[9.456603050231934, 5.028219699859619, 7.063089847564697, 6.071815490722656, 5.3074727058410645]
第二个文档前后分数变化过大

Beijing Academy of Artificial Intelligence org

You can use our newest reranker model: BAAI/bge-reranker-v2-m3

ok, I will change the reranker model to newest one to test.

Sign up or log in to comment