Repository: wandb/wandb

Issue:
Runs returned by `wandb.Api` contain duplicates not present on the web app
### Describe the bug

<!--- Description of the issue below  -->
I want to do some analysis of a large group of runs, but the runs returned contained duplicates instead of some of the runs
<!--- A minimal code snippet between the quotes below  -->
```python
import wandb
import pandas as pd
import numpy as np

api = wandb.Api()
filter_dict = {
    "group": "fnornn-test-seeds-40hz-fixed",
}
runs = api.runs("iir-modal/physmodjax", filter_dict)
print(len(runs))  #18962, same as in web app
```
```python
data = []
for run in runs:
    data.append(
        {
            "seed": run.config["seed"],
            "num_steps_train": run.config["datamodule"]["num_steps_train"],
            "train_loss": run.summary["train loss"],
            "run_name": run.name,
        },
    )
print(len(data))  #18962
```

```python
df = pd.DataFrame(data)
df_indx = df.set_index(["seed", "num_steps_train"])
indx_list = np.where(df_indx.index.duplicated()) 
print(len(indx_list[0])) # 87, this is bad

indx_dup = df_indx.index[indx_list]
for i in range(len(indx_dup)):
    print(df_indx.loc[indx_dup[i]])
```
![image](https://github.com/wandb/wandb/assets/32675952/a30d0bc6-ddf5-4812-a34f-ec5973b3cbeb)

This runs are not duplicated in the web app, for example
![image](https://github.com/wandb/wandb/assets/32675952/61c1e70d-cb10-4cbb-9d89-5aa1f806596a)

When looking at the runs,
```python
print(runs[100])
print(runs[99])
```
```bash
<Run iir-modal/physmodjax/4kp6z7va (finished)>
<Run iir-modal/physmodjax/4kp6z7va (finished)>
```

There is nothing special about either the duplicated runs, or the 87 that are missing in their place, except that the indexes are suspicioulsly round in general, and the pairs of duplicates are all close to each other:
```python
for ind in indx_list[0]:
    # print(df.loc[ind])
    ind_rep = df.loc[df["run_name"] == df.loc[ind, "run_name"]].index
    print(
        f"runs[{ind_rep[0]}]: {runs[ind_rep[0]]} and runs[{ind_rep[1]}]: {runs[ind_rep[1]]}"
    )
```
```bash
runs[99]: <Run iir-modal/physmodjax/4kp6z7va (finished)> and runs[100]: <Run iir-modal/physmodjax/4kp6z7va (finished)>
runs[449]: <Run iir-modal/physmodjax/vhcasx2u (finished)> and runs[450]: <Run iir-modal/physmodjax/vhcasx2u (finished)>
runs[497]: <Run iir-modal/physmodjax/z6k0i9jq (finished)> and runs[500]: <Run iir-modal/physmodjax/z6k0i9jq (finished)>
runs[499]: <Run iir-modal/physmodjax/ujk0lwhk (finished)> and runs[501]: <Run iir-modal/physmodjax/ujk0lwhk (finished)>
runs[598]: <Run iir-modal/physmodjax/lrug2iut (finished)> and runs[600]: <Run iir-modal/physmodjax/lrug2iut (finished)>
runs[699]: <Run iir-modal/physmodjax/6ztlfdqu (finished)> and runs[701]: <Run iir-modal/physmodjax/6ztlfdqu (finished)>
runs[747]: <Run iir-modal/physmodjax/m0f8fyq3 (finished)> and runs[750]: <Run iir-modal/physmodjax/m0f8fyq3 (finished)>
runs[749]: <Run iir-modal/physmodjax/rahnut8i (finished)> and runs[751]: <Run iir-modal/physmodjax/rahnut8i (finished)>
runs[798]: <Run iir-modal/physmodjax/asbcv093 (finished)> and runs[800]: <Run iir-modal/physmodjax/asbcv093 (finished)>
runs[849]: <Run iir-modal/physmodjax/9e478eit (finished)> and runs[850]: <Run iir-modal/physmodjax/9e478eit (finished)>
runs[949]: <Run iir-modal/physmodjax/v2a4vxbh (finished)> and runs[951]: <Run iir-modal/physmodjax/v2a4vxbh (finished)>
runs[997]: <Run iir-modal/physmodjax/4ltf1ku8 (finished)> and runs[1000]: <Run iir-modal/physmodjax/4ltf1ku8 (finished)>
runs[1148]: <Run iir-modal/physmodjax/7rfifmar (finished)> and runs[1150]: <Run iir-modal/physmodjax/7rfifmar (finished)>
runs[1449]: <Run iir-modal/physmodjax/43vznps1 (finished)> and runs[1450]: <Run iir-modal/physmodjax/43vznps1 (finished)>
runs[1499]: <Run iir-modal/physmodjax/i720ra6r (finished)> and runs[1500]: <Run iir-modal/physmodjax/i720ra6r (finished)>
runs[1599]: <Run iir-modal/physmodjax/2enq3lb8 (finished)> and runs[1600]: <Run iir-modal/physmodjax/2enq3lb8 (finished)>
runs[1749]: <Run iir-modal/physmodjax/1xdoe710 (finished)> and runs[1751]: <Run iir-modal/physmodjax/1xdoe710 (finished)>
runs[1999]: <Run iir-modal/physmodjax/fkfayt36 (finished)> and runs[2000]: <Run iir-modal/physmodjax/fkfayt36 (finished)>
runs[2099]: <Run iir-modal/physmodjax/be1hpp9n (finished)> and runs[2100]: <Run iir-modal/physmodjax/be1hpp9n (finished)>
runs[2148]: <Run iir-modal/physmodjax/hjidwkdm (finished)> and runs[2151]: <Run iir-modal/physmodjax/hjidwkdm (finished)>
runs[2199]: <Run iir-modal/physmodjax/aj4sm9r0 (finished)> and runs[2202]: <Run iir-modal/physmodjax/aj4sm9r0 (finished)>
runs[2249]: <Run iir-modal/physmodjax/5tvbwwbf (finished)> and runs[2251]: <Run iir-modal/physmodjax/5tvbwwbf (finished)>
runs[2298]: <Run iir-modal/physmodjax/en65g0ld (finished)> and runs[2301]: <Run iir-modal/physmodjax/en65g0ld (finished)>
runs[2399]: <Run iir-modal/physmodjax/nvtpml5b (finished)> and runs[2400]: <Run iir-modal/physmodjax/nvtpml5b (finished)>
runs[2398]: <Run iir-modal/physmodjax/sxsee24x (finished)> and runs[2401]: <Run iir-modal/physmodjax/sxsee24x (finished)>
runs[2498]: <Run iir-modal/physmodjax/sy9i81b8 (finished)> and runs[2500]: <Run iir-modal/physmodjax/sy9i81b8 (finished)>
runs[2699]: <Run iir-modal/physmodjax/q2dzsmnq (finished)> and runs[2701]: <Run iir-modal/physmodjax/q2dzsmnq (finished)>
runs[2749]: <Run iir-modal/physmodjax/ngizfdbh (finished)> and runs[2750]: <Run iir-modal/physmodjax/ngizfdbh (finished)>
runs[2799]: <Run iir-modal/physmodjax/gvkigc5y (finished)> and runs[2800]: <Run iir-modal/physmodjax/gvkigc5y (finished)>
runs[2998]: <Run iir-modal/physmodjax/sk37dspl (finished)> and runs[3000]: <Run iir-modal/physmodjax/sk37dspl (finished)>
runs[3349]: <Run iir-modal/physmodjax/q6o6im9c (finished)> and runs[3350]: <Run iir-modal/physmodjax/q6o6im9c (finished)>
runs[3449]: <Run iir-modal/physmodjax/0q9001ir (finished)> and runs[3450]: <Run iir-modal/physmodjax/0q9001ir (finished)>
runs[3549]: <Run iir-modal/physmodjax/iyh34hh0 (finished)> and runs[3550]: <Run iir-modal/physmodjax/iyh34hh0 (finished)>
runs[3699]: <Run iir-modal/physmodjax/9v289xww (finished)> and runs[3701]: <Run iir-modal/physmodjax/9v289xww (finished)>
runs[3897]: <Run iir-modal/physmodjax/fy3vxfmi (finished)> and runs[3900]: <Run iir-modal/physmodjax/fy3vxfmi (finished)>
runs[4099]: <Run iir-modal/physmodjax/ysn4j48x (finished)> and runs[4100]: <Run iir-modal/physmodjax/ysn4j48x (finished)>
runs[4399]: <Run iir-modal/physmodjax/8yp0g96x (finished)> and runs[4401]: <Run iir-modal/physmodjax/8yp0g96x (finished)>
runs[4499]: <Run iir-modal/physmodjax/cir7zz23 (finished)> and runs[4500]: <Run iir-modal/physmodjax/cir7zz23 (finished)>
runs[4749]: <Run iir-modal/physmodjax/yrj5xuc6 (finished)> and runs[4750]: <Run iir-modal/physmodjax/yrj5xuc6 (finished)>
runs[4849]: <Run iir-modal/physmodjax/d1vnuoyu (finished)> and runs[4850]: <Run iir-modal/physmodjax/d1vnuoyu (finished)>
runs[4949]: <Run iir-modal/physmodjax/nyv06wht (finished)> and runs[4950]: <Run iir-modal/physmodjax/nyv06wht (finished)>
runs[4998]: <Run iir-modal/physmodjax/md7dxhfp (finished)> and runs[5001]: <Run iir-modal/physmodjax/md7dxhfp (finished)>
runs[5048]: <Run iir-modal/physmodjax/sx1829pq (finished)> and runs[5050]: <Run iir-modal/physmodjax/sx1829pq (finished)>
runs[5299]: <Run iir-modal/physmodjax/zse5hsqg (finished)> and runs[5300]: <Run iir-modal/physmodjax/zse5hsqg (finished)>
runs[5349]: <Run iir-modal/physmodjax/jgsfd6kn (finished)> and runs[5351]: <Run iir-modal/physmodjax/jgsfd6kn (finished)>
runs[5398]: <Run iir-modal/physmodjax/qvy2wosy (finished)> and runs[5400]: <Run iir-modal/physmodjax/qvy2wosy (finished)>
runs[5648]: <Run iir-modal/physmodjax/h0u79suh (finished)> and runs[5651]: <Run iir-modal/physmodjax/h0u79suh (finished)>
runs[5999]: <Run iir-modal/physmodjax/skksbe5k (finished)> and runs[6000]: <Run iir-modal/physmodjax/skksbe5k (finished)>
runs[5998]: <Run iir-modal/physmodjax/boxd5i0q (finished)> and runs[6001]: <Run iir-modal/physmodjax/boxd5i0q (finished)>
runs[6099]: <Run iir-modal/physmodjax/mwfq41nm (finished)> and runs[6100]: <Run iir-modal/physmodjax/mwfq41nm (finished)>
runs[6098]: <Run iir-modal/physmodjax/uiinqnl5 (finished)> and runs[6104]: <Run iir-modal/physmodjax/uiinqnl5 (finished)>
runs[6249]: <Run iir-modal/physmodjax/x2xi2pln (finished)> and runs[6251]: <Run iir-modal/physmodjax/x2xi2pln (finished)>
runs[6399]: <Run iir-modal/physmodjax/7a977w1z (finished)> and runs[6400]: <Run iir-modal/physmodjax/7a977w1z (finished)>
runs[6397]: <Run iir-modal/physmodjax/gbkm4c81 (finished)> and runs[6401]: <Run iir-modal/physmodjax/gbkm4c81 (finished)>
runs[6398]: <Run iir-modal/physmodjax/y2o8y9bz (finished)> and runs[6402]: <Run iir-modal/physmodjax/y2o8y9bz (finished)>
runs[6448]: <Run iir-modal/physmodjax/sahtepl9 (finished)> and runs[6452]: <Run iir-modal/physmodjax/sahtepl9 (finished)>
runs[6649]: <Run iir-modal/physmodjax/6z6vk03m (finished)> and runs[6650]: <Run iir-modal/physmodjax/6z6vk03m (finished)>
runs[6749]: <Run iir-modal/physmodjax/yr89n9w8 (finished)> and runs[6751]: <Run iir-modal/physmodjax/yr89n9w8 (finished)>
runs[6798]: <Run iir-modal/physmodjax/v1rpos7j (finished)> and runs[6800]: <Run iir-modal/physmodjax/v1rpos7j (finished)>
runs[6849]: <Run iir-modal/physmodjax/aqj6ys44 (finished)> and runs[6851]: <Run iir-modal/physmodjax/aqj6ys44 (finished)>
runs[6896]: <Run iir-modal/physmodjax/gkq37u0d (finished)> and runs[6900]: <Run iir-modal/physmodjax/gkq37u0d (finished)>
runs[6949]: <Run iir-modal/physmodjax/48c43ae6 (finished)> and runs[6951]: <Run iir-modal/physmodjax/48c43ae6 (finished)>
runs[6999]: <Run iir-modal/physmodjax/g6xipn8w (finished)> and runs[7000]: <Run iir-modal/physmodjax/g6xipn8w (finished)>
runs[7048]: <Run iir-modal/physmodjax/idtyf4u8 (finished)> and runs[7050]: <Run iir-modal/physmodjax/idtyf4u8 (finished)>
runs[7149]: <Run iir-modal/physmodjax/rfrg4lve (finished)> and runs[7150]: <Run iir-modal/physmodjax/rfrg4lve (finished)>
runs[7297]: <Run iir-modal/physmodjax/hga92bca (finished)> and runs[7300]: <Run iir-modal/physmodjax/hga92bca (finished)>
runs[7496]: <Run iir-modal/physmodjax/4mt7w6co (finished)> and runs[7500]: <Run iir-modal/physmodjax/4mt7w6co (finished)>
runs[7497]: <Run iir-modal/physmodjax/gm7quxia (finished)> and runs[7503]: <Run iir-modal/physmodjax/gm7quxia (finished)>
runs[7499]: <Run iir-modal/physmodjax/tvo6qipq (finished)> and runs[7506]: <Run iir-modal/physmodjax/tvo6qipq (finished)>
runs[7549]: <Run iir-modal/physmodjax/yw17kf45 (finished)> and runs[7550]: <Run iir-modal/physmodjax/yw17kf45 (finished)>
runs[7599]: <Run iir-modal/physmodjax/4zggqle2 (finished)> and runs[7603]: <Run iir-modal/physmodjax/4zggqle2 (finished)>
runs[8099]: <Run iir-modal/physmodjax/btfalbbj (finished)> and runs[8100]: <Run iir-modal/physmodjax/btfalbbj (finished)>
runs[8149]: <Run iir-modal/physmodjax/cdstls5y (finished)> and runs[8150]: <Run iir-modal/physmodjax/cdstls5y (finished)>
runs[8249]: <Run iir-modal/physmodjax/mgwt8lrf (finished)> and runs[8250]: <Run iir-modal/physmodjax/mgwt8lrf (finished)>
runs[8298]: <Run iir-modal/physmodjax/wuxyy5xe (finished)> and runs[8300]: <Run iir-modal/physmodjax/wuxyy5xe (finished)>
runs[8349]: <Run iir-modal/physmodjax/xanfgzqu (finished)> and runs[8351]: <Run iir-modal/physmodjax/xanfgzqu (finished)>
runs[8348]: <Run iir-modal/physmodjax/2gvnxbtb (finished)> and runs[8352]: <Run iir-modal/physmodjax/2gvnxbtb (finished)>
runs[8346]: <Run iir-modal/physmodjax/pm61152y (finished)> and runs[8353]: <Run iir-modal/physmodjax/pm61152y (finished)>
runs[8398]: <Run iir-modal/physmodjax/ax4djvc6 (finished)> and runs[8402]: <Run iir-modal/physmodjax/ax4djvc6 (finished)>
runs[8443]: <Run iir-modal/physmodjax/9b1oj8vf (finished)> and runs[8450]: <Run iir-modal/physmodjax/9b1oj8vf (finished)>
runs[8598]: <Run iir-modal/physmodjax/abnwba4x (finished)> and runs[8600]: <Run iir-modal/physmodjax/abnwba4x (finished)>
runs[8897]: <Run iir-modal/physmodjax/5o3frsvu (finished)> and runs[8900]: <Run iir-modal/physmodjax/5o3frsvu (finished)>
runs[9099]: <Run iir-modal/physmodjax/s2k8e9a7 (finished)> and runs[9100]: <Run iir-modal/physmodjax/s2k8e9a7 (finished)>
runs[9149]: <Run iir-modal/physmodjax/vuyhs0ce (finished)> and runs[9150]: <Run iir-modal/physmodjax/vuyhs0ce (finished)>
runs[9199]: <Run iir-modal/physmodjax/p5ogebi3 (finished)> and runs[9200]: <Run iir-modal/physmodjax/p5ogebi3 (finished)>
runs[9248]: <Run iir-modal/physmodjax/fns5zgqp (finished)> and runs[9250]: <Run iir-modal/physmodjax/fns5zgqp (finished)>
runs[9299]: <Run iir-modal/physmodjax/zggfogud (finished)> and runs[9301]: <Run iir-modal/physmodjax/zggfogud (finished)>
```

This behaviour is consistent even after `api.flush()` and updating to 0.16.0 

There is nothing recorded in `/tmp/debug-cli.carlos.log`. I don't know if the `wandb.Api` object keeps a log anywhere, haven't been able to find anything in the docs.


### Additional Files

_No response_

### Environment

WandB version: 0.15.12 and 0.16.0

OS:
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal

Python version: 3.9.18 

Versions of relevant libraries:
pandas==2.1.1



### Additional Context

_No response_


Discussion:
Hi @cdelavegamartin thank you for reporting this issue.  Can you please try to run your code again without applying any filtering by removing `filter_dict` argument? It seems you're checking if you have any duplicate entries based on run.config["seed"], is that correct? and is there any chance you've created multiple runs with same `seed`?
Hi @cdelavegamartin We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
Hi, thanks for the help, wasn't able to test this earlier.
> Hi @cdelavegamartin thank you for reporting this issue. Can you please try to run your code again without applying any filtering by removing `filter_dict` argument? It seems you're checking if you have any duplicate entries based on run.config["seed"], is that correct? and is there any chance you've created multiple runs with same `seed`?

I'm checking duplicates based on a multiindex: `run.config["seed"]` and `run.config["datamodule"]["num_steps_train"]`. This tuple is unique for each run in this group (there are many runs for each seed, and many runs for each `num_steps_train`, but a single one for each combination). This the case in the web interface, as shown above for one example `dandy-dragon-38029`. Not all runs in this project have either a group assigned, or the same config fields, so I've simplified the test further:

```python
api = wandb.Api()
api.flush()
runs_all = api.runs("iir-modal/physmodjax")
len(runs_all) # 21383
names = []
for run in runs_all:
    names.append(run.name)
duplicates = []
for name in set(names):
    if names.count(name) > 1:
        duplicates.append(name)

print(len(names)-len(set(names))) # 102

for name in duplicates:
    indices = [i for i, x in enumerate(names) if x == name]
    print(f"name: {name}, indices: {indices}")
    for i in indices:
        print(f"run: {runs_all[i]}")
```
![image](https://github.com/wandb/wandb/assets/32675952/21321f41-ee53-4060-a5a5-fefb846acd8b)
![image](https://github.com/wandb/wandb/assets/32675952/3fb91724-d2a7-4227-8418-8c244a14a26e)
![image](https://github.com/wandb/wandb/assets/32675952/70a686ff-b8f7-4007-8b0a-3271a7d6cbe7)

As you can see there are a few (~0.5%) runs that are repeated, when accesing through the API. 



If the duplicates were in addition to all the runs, it would be annoying, but easily fixable on my side. As they are not (there are runs that don't get loaded  through the API) this is a significant problem, as I'm trying to get statistics for large sweeps.
Also, the duplicates when filtering by group vs taking all of them are different (but repeatable), which together with how closely spaced they are, and always around round number makes me strongly suspect in an issue with batching or similar in the API.
```python
known_dup = "dandy-dragon-38029" # One of the duplicates when filtering by group
assert known_dup in duplicates # Fails
``` 
Hi @cdelavegamartin thank you for the detailed information. It seems than `len(runs_all)` is the same as the one shown in the screenshot above Runs search (`21383`). Therefore it looks like the API correctly returns what's in the App too. The unique identifiers in a project are the run ids. Since I don't have a reproducible example, can you please try to run your code once again by modifying this part:

```
runs_all = api.runs("iir-modal/physmodjax")
len(runs_all) # 21383
ids = []
for run in runs_all:
    ids.append(run.id)
```

Do you still get any duplicates returns? 

Hi @thanos-wandb, it does return the same number of runs indeed, but some of those runs are duplicates (and therefore some are missing in their place), that is in essence the problem. I don't set the names manually, and for what I understand wandb should be generating unique names. This does seem to be the case (See for example  `earnest-durian-31819` above) in the web interface, whereas both elements in the runs_all list contain the same run object:
![image](https://github.com/wandb/wandb/assets/32675952/2f09a9b6-3e39-4f55-8676-a2c22e259274)

I ran the code with your modification so that this is more explicit:
![image](https://github.com/wandb/wandb/assets/32675952/efebc6b7-e193-43dc-9b7c-2f805cbb4704)

Note that in the meanwhile I've added around 2k more runs, but the number of duplicates has almost doubled, and again, these are different to the ones I was getting duplicated before, and the runs are not duplicated in the web interface (now more evident because they wouldn't be able to share the same ID). The runs have been logged mostly with wandb sync after running them offline, but many of them where also recorded online. That doesn't seem to matter as to the runs affected,  and is worth bearing in mind that as I showed before, the runs affected change depending on the number of runs I'm accessing through the API (either because the total number of runs changes, or because of filtering)

Is there anything I could provide for reproducibility? I can provide access to the original code (it will take a few hours/days to rerun at least some of the experiments if you wish to), and of course, happy for someone to have a look at the project in WandB for debugging. I haven't tried to build a minimal example, I could try to create a new project and log a large amount of dummy runs? This would use up valuable compute time at my uni hpc, so I would prefer if it could be avoided.

Hi @cdelavegamartin thanks so much for the additional information here, this is very helpful. It seems the error to be with the API returning paginated results, rather that with our backend - as the runs show up as unique in the App side. I am logging in a project thousands of dummy runs to try and reproduce this behavior - and I will let you know if there's any other information needed on your side, but this should suffice for now. Thanks once again, and I will keep you updated here!
Hi @cdelavegamartin I have tried the same code as in your notebook above in a project of mine with 25k runs, and it seems to not return any duplicates. I am wondering if you're noticing the same issue in other projects too? also, would it be possible to try in another environment too? just to exclude the possibility this is related with Jupyter notebook environment. Thank you!
Hi @cdelavegamartin we wanted to follow up with you regarding your support request as we have not heard back from you. Could you please check if you're seeing this issue in any other projects? 
Hi @cdelavegamartin, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!
@thanos-wandb Apologies for tagging you on a closed issue but I've just seen the same problem on another project. The total number of runs returned by a query (571 runs in this case) is correct but one of them is duplicated, so there are only 570 unique runs returned and one of the runs that _should_ have been returned is missing.

It's maybe related to paging somehow as changing `per_page` in the runs args changes the behaviour:

```python
runs = api.runs(
    "<my_project_name>",
    filters={
        "created_at": {"$gt": "2024-01-01T"},
        "State": "finished",
        "jobType": "<my_job_type>",
    },
    per_page=per_page,  # replaced with values below
)
```

#### per_page=10 (wrong result)

```
len(runs)=571
len(set(r.id for r in runs))=569
```

#### per_page=50 (wrong result)

```
len(train_runs)=571
len(set(r.id for r in train_runs))=570
```

#### per_page=60 (correct result)

```
len(runs)=571
len(set(r.id for r in runs))=571
```

#### per_page=100 (correct result)

```
len(runs)=571
len(set(r.id for r in runs))=571
```



Hi @jack89roberts thank you for reporting this issue as well. Are you able to consistently reproduce this issue? I am wondering if it's related to applied rate limits as it wasn't reproducible on my end. Would it be please possible to explicitly specify the next page of paginated runs as in the code snippet below?

```
import wandb
api = wandb.Api()
runs_iterator = api.runs(path, per_page=50)
for _ in range(runs_iterator.length):
  runs = runs_iterator.next()
  print(len(set(r.id for r in runs)))
```
Thanks for your reply @thanos-wandb , I tried your snipped but it doesn't work for me - it seems `runs_iterator.length` is not always immediately set and can be `None` initially, and also (for me at least) `runs_iterator.next()` returns a single run each time (I think the intent above was for it to be a page of runs?)

Does this similar snippet do what you were looking for?

```python
import wandb
api = wandb.Api()
runs_iterator = api.runs(path, per_page=50)
runs = []
for _ in range(len(runs_iterator)):
    new = runs_iterator.next()
    dup = new.id in runs
    runs.append(new.id)
    if dup:
        print(len(runs))
```

I get duplicates at iterations 501, 701, 801, 1002, 1451, 1551, 1552, and 1651 (with no filters applied there are 3333 runs in this project, rather than the 571 I mentioned in the previous comment).

re reproducibility - we've seen it consistently today between two users (querying the same project but running off different machines & networks, if that's relevant for rate limits etc.)
If I add some print statements in `Paginator.load_page` and `Runs.convert_objects` I can see duplicate IDs between pages, e.g.

```
Query variables: {'project': '...', 'entity': '...', 'order': '-created_at', 'filters': '{...}', 'perPage': 50, 'cursor': 'YXJyYXljb25uZWN0aW9uOjE5OQ=='}
-> Returns 49 runs and then run id dibltvkk
```

is followed by

```
Query variables: {'project': '...', 'entity': '..., 'order': '-created_at', 'filters': '{...}', 'perPage': 50, 'cursor': 'YXJyYXljb25uZWN0aW9uOjI0OQ=='}
-> Returns run id dibltvkk and then 49 runs
```

But this isn't for every page (and it's not necessarily always the last run of one page and the first run of the next either).

Changing the `order` value can change how many duplicates there are (and where they are) too. I'm guessing it's something to do with the cursor/consistent ordering between pages, but not sure how to debug it further.
Hi there,


> Hi @cdelavegamartin I have tried the same code as in your notebook above in a project of mine with 25k runs, and it seems to not return any duplicates. I am wondering if you're noticing the same issue in other projects too? also, would it be possible to try in another environment too? just to exclude the possibility this is related with Jupyter notebook environment. Thank you!

At the moment I haven't got any other projects apart from a few toy ones on wandb, so haven't been able to test this in a different project.
 
I ran the following as a script, instead of a notebook:
```python
import wandb

api = wandb.Api()
api.flush()
runs_all = api.runs("iir-modal/physmodjax")
len(runs_all)

run_ids = []
for run in runs_all:
    run_ids.append(run.id)

print(f"Number of runs: {len(run_ids)}")

# Get a list with all the duplicates
duplicates = []
for run_id in set(run_ids):
    if run_ids.count(run_id) > 1:
        duplicates.append(run_id)

# Print the number of run_ids and the number of unique run_ids
print(f"Number of runs: {len(run_ids)}")
print(f"Number of non-unique id: {len(run_ids)-len(set(run_ids))}")

# Print the first n duplicates
n_dup = 20
print(f"First {n_dup} duplicates:")
for dup in duplicates[:n_dup]:
    inds_dup = [i for i, x in enumerate(run_ids) if x == dup]
    print(f"Duplicate id: {dup}, indices: {inds_dup}")
    for i in inds_dup:
        run = runs_all[i]
        print(f"run: {run} run id: {run.id}, name: {run.name}")
```

```bash
$:~/projects/physmodjax$ python examples/test_wandb_dup.py 
Number of runs: 23623
Number of runs: 23623
Number of non-unique id: 1
First 20 duplicates:
Duplicate id: qmtj4pgz, indices: [1599, 1605]
run: <Run iir-modal/physmodjax/qmtj4pgz (finished)> run id: qmtj4pgz, name: glad-dawn-39817
run: <Run iir-modal/physmodjax/qmtj4pgz (finished)> run id: qmtj4pgz, name: glad-dawn-39817
```

Only getting one duplicate this time
Is there any difference between how it would run in a notebook or a script?
Last couple months I have been focusing on other aspects, but I will soon be running large sweeps again, so it would be fantastic if we could find a solution 
![image](https://github.com/wandb/wandb/assets/32675952/816cc94c-84cd-46e2-85bd-013551d75eb3)
Running in a notebook again
I'm running into the exact same issue when gathering runs from `api.sweep`. I don't get the duplicates often, or consistently across projects. It's always the same 1 or 2 sweeps that do not export correctly and give duplicates.

1 or 2 duplicates might not seem problematic, but errors in the experiment bookkeeping makes me distrust any of my results.

For example,
```python
sweep = api.sweep(path)
all_runs = list(sweep.runs)
all_runs[238:242]
>> [<Run joery/FineTune/do92cdk4 (finished)>,
 <Run joery/FineTune/chmuj6zr (finished)>,
 <Run joery/FineTune/chmuj6zr (finished)>,
 <Run joery/FineTune/rseajrc8 (finished)>]
```

EDIT: Wanted to add, when I approached the export through `api.runs` I did not get duplicates... 
Hi @jack89roberts @cdelavegamartin @joeryjoery sorry for the late reply here, this ticket was closed and had slipped through the cracks. We are investigating this behavior currently and I will keep you updated once we have a fix for this. 

Resolving patch:
diff --git a/wandb/apis/public/api.py b/wandb/apis/public/api.py
--- a/wandb/apis/public/api.py
+++ b/wandb/apis/public/api.py
@@ -757,7 +757,7 @@ def runs(
         self,
         path: Optional[str] = None,
         filters: Optional[Dict[str, Any]] = None,
-        order: str = "-created_at",
+        order: str = "+created_at",
         per_page: int = 50,
         include_sweeps: bool = True,
     ):


Test patch: