Spaces:
Running
on
CPU Upgrade
Unable to Query the Leaderboard via Gradio Client.
I made a very recent clone of the Leaderboard and when I use client like this:
space = "https://felixz-open-llm-leaderboard2.hf.space/"
#space = "http://127.0.0.1:7862/"
client = Client(space)
json_data = client.predict( fn_index=6)
Output:
Loaded as API: https://felixz-open-llm-leaderboard2.hf.space/ ✔
Traceback (most recent call last):
File "C:\Users\felix\dev\meta_llm_leaderboard\scripts\hf_llm_leaderboard.py", line 16, in
json_data = client.predict( fn_index=6)
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\site-packages\gradio_client\client.py", line 291, in predict
return self.submit(*args, api_name=api_name, fn_index=fn_index).result()
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\site-packages\gradio_client\client.py", line 1120, in result
return super().result(timeout=timeout)
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\concurrent\futures_base.py", line 458, in result
return self.__get_result()
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\concurrent\futures_base.py", line 403, in __get_result
raise self._exception
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\site-packages\gradio_client\client.py", line 793, in _inner
predictions = _predict(*data)
File "C:\Users\felix\anaconda3\envs\useful-python-scripts\lib\site-packages\gradio_client\client.py", line 824, in _predict
raise ValueError(result["error"])
ValueError: None
On the Logs side this is an exception logged:
43.05it/s]
99%|█████████▉| 1243/1257 [00:55<00:00, 42.45it/s]
100%|█████████▉| 1251/1257 [00:55<00:00, 48.90it/s]
100%|██████████| 1257/1257 [00:55<00:00, 22.53it/s]
Pulling changes for the evaluation queue.
Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/gradio/routes.py", line 401, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1305, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/home/user/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1239, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/home/user/.local/lib/python3.10/site-packages/gradio/components.py", line 2995, in postprocess
"data": Dataframe.__process_markdown(
File "/home/user/.local/lib/python3.10/site-packages/gradio/components.py", line 3051, in __process_markdown
data[i][j] = cls.markdown_parser.render(data[i][j])
File "/home/user/.local/lib/python3.10/site-packages/markdown_it/main.py", line 267, in render
return self.renderer.render(self.parse(src, env), self.options, env)
File "/home/user/.local/lib/python3.10/site-packages/markdown_it/main.py", line 250, in parse
raise TypeError(f"Input data should be a string, not {type(src)}")
TypeError: Input data should be a string, not <class 'float'>
I see the same client error when I query the public main leaderboard but of course, I have no access to logs there.
Hi
@felixz
Thank you for your issue, I pinged the gradio team!
However, why are you trying to query it like so?
All the data is already in public datasets, and your https://felixz-open-llm-leaderboard2.hf.space/ seems to work fine.
@clefourrier I am trying to get the data from the official LLM leaderboard space. I will download all the rows as they show on the leaderboard and store them to csv so I can process them to make graphs and so on for my leaderboard project. I think a few others are also doing that as well. I run a Python script locally that uses Gradio client to pull in the data as a request as I showed.
If you want the official data, it all comes from the results dataset every hour - you might as well grab it frem there directly :)
I am trying to use results repo but it is complicated by the fact it assumes Linux filesystem and on my windows pc i am getting errors checking out files with invalid characters in path for windows. The gradio api used to abstract everything away nicely but it stopped suddenly working. It is probably some kind of data issue.
Did you try to simply use a git clone to get the results repository, or are you using a ds = datasets.load_dataset(?
Yes, I am just doing the same code that Leaderboard app is using which is doing a git lfs clone. The problem is because of "results...json" files have a semicolon in the filename the git clone command fails on Windows system. On Linux, it will be fine. Semicolon is not a valid character that can be in the filename on Windows. If you just replace the semicolon with a dash it will work for anyone working with the results repo on Windows machine.
Here is is how my code works if that helps:
https://github.com/felix-zaslavskiy/meta_llm_leaderboard/blob/d1fa7507a877997a37ee095981f7c72f0af47f17/scripts/hf_llm_leaderboard.py#L17
Good catch, thank you!
Hi! I think this (using gradio through the API) should be fixed now, please tell me how it works for you!
works now thanks
Cool, closing the issue!