Repository: saketkc/pysradb

Issue:
[BUG] ValueError when using SraSearch to query
**Describe the bug**
Using `SraSearch` with `verbosity>=2` and a large query raises a ValueError when setting `self.df["pmid"] = list(uids)` (https://github.com/saketkc/pysradb/blob/c23d4a769543d05a0f002d1b28c985da5963573f/pysradb/search.py#L776) because the size of the underlying dataframe seems to vary.

The following error is raised:
```python
ValueError                                Traceback (most recent call last)
<ipython-input-1-d19c33e66199> in <module>
      1 instance = SraSearch(verbosity=3, return_max=max_query_num, query=query, platform='illumina')
----> 2 instance.search()
      3 df_search = instance.get_df()

/pysradb/search.py in search(self)
    774             self._format_result()
    775             if self.verbosity >= 2:
--> 776                 self.df["pmid"] = list(uids)
    777         except requests.exceptions.Timeout:
    778             sys.exit(f"Connection to the server has timed out. Please retry.")

/pandas/core/frame.py in __setitem__(self, key, value)
   3038         else:
   3039             # set column
-> 3040             self._set_item(key, value)
   3041
   3042     def _setitem_slice(self, key: slice, value):

/pandas/core/frame.py in _set_item(self, key, value)
   3114         """
   3115         self._ensure_valid_index(value)
-> 3116         value = self._sanitize_column(key, value)
   3117         NDFrame._set_item(self, key, value)
   3118

/pandas/core/frame.py in _sanitize_column(self, key, value, broadcast)
   3762
   3763             # turn me into an ndarray
-> 3764             value = sanitize_index(value, self.index)
   3765             if not isinstance(value, (np.ndarray, Index)):
   3766                 if isinstance(value, list) and len(value) > 0:

/pandas/core/internals/construction.py in sanitize_index(data, index)
    745     """
    746     if len(data) != len(index):
--> 747         raise ValueError(
    748             "Length of values "
    749             f"({len(data)}) "

ValueError: Length of values (86768) does not match length of index (86721)
```

Multiple runs yield slightly different error messages:
```python
ValueError: Length of values (86768) does not match length of index (86760)
```

It seems like the index length is varying for some reason.

**To Reproduce**
Execute the following code:
```
from pysradb.search import SraSearch

max_query_num = 1_000_000
query = 'txid2697049[Organism:noexp] AND ("filetype cram"[Properties] OR "filetype bam"[Properties] OR "filetype fastq"[Properties])'

instance = SraSearch(verbosity=2, return_max=max_query_num, query=query, platform='illumina')
instance.search()
df_search = instance.get_df()
```


**Desktop:**
 - OS: `Linux`
 - Python version: `3.8.5`
 - pysradb version: `0.11.2-dev0`


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=h1) Report
> Merging [#92](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=desc) (989155f) into [master](https://codecov.io/gh/saketkc/pysradb/commit/733c938d4328c4b6ccf3394f3b107c285468d2d9?el=desc) (733c938) will **increase** coverage by `0.22%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/saketkc/pysradb/pull/92/graphs/tree.svg?width=650&height=150&src=pr&token=w1fPTXpzK2)](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #92      +/-   ##
==========================================
+ Coverage   54.19%   54.41%   +0.22%     
==========================================
  Files           7        7              
  Lines        1777     1777              
==========================================
+ Hits          963      967       +4     
+ Misses        814      810       -4     
```


| [Impacted Files](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pysradb/search.py](https://codecov.io/gh/saketkc/pysradb/pull/92/diff?src=pr&el=tree#diff-cHlzcmFkYi9zZWFyY2gucHk=) | `79.81% <100.00%> (ø)` | |
| [pysradb/sraweb.py](https://codecov.io/gh/saketkc/pysradb/pull/92/diff?src=pr&el=tree#diff-cHlzcmFkYi9zcmF3ZWIucHk=) | `80.76% <0.00%> (+0.80%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=footer). Last update [733c938...989155f](https://codecov.io/gh/saketkc/pysradb/pull/92?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/pysradb/search.py b/pysradb/search.py
index a0a582c..613c44b 100644
--- a/pysradb/search.py
+++ b/pysradb/search.py
@@ -1,6 +1,5 @@
 """This file contains the search classes for the search feature.
 """
-import numpy as np
 import os
 import re
 import requests
@@ -78,9 +77,19 @@ class QuerySearch:
     -------
     get_df()
         Returns the dataframe storing this search result.
+
     search()
         Executes the search.
 
+    show_result_statistics()
+        Shows summary information about search results.
+
+    visualise_results()
+        Generate graphs that visualise the search results.
+
+    get_plot_objects():
+        Get the plot objects for plots generated.
+
     """
 
     def __init__(
@@ -682,6 +691,18 @@ class SraSearch(QuerySearch):
         sends the user query via requests to NCBI Entrez API and returns
         search results as a pandas dataframe.
 
+    show_result_statistics()
+        Shows summary information about search results.
+
+    visualise_results()
+        Generate graphs that visualise the search results.
+
+    get_plot_objects():
+        Get the plot objects for plots generated.
+
+    get_uids():
+        Get NCBI uids retrieved during this search query.
+
     _format_query_string()
         formats the input user query into a string
 
@@ -732,6 +753,7 @@ class SraSearch(QuerySearch):
         )
         self.entries = {}
         self.number_entries = 0
+        self.uids = []
 
     def search(self):
         # Step 1: retrieves the list of uids that satisfies the input
@@ -744,21 +766,21 @@ class SraSearch(QuerySearch):
                 timeout=SEARCH_REQUEST_TIMEOUT,
             )
             r.raise_for_status()
-            uids = r.json()["esearchresult"]["idlist"]
+            self.uids = r.json()["esearchresult"]["idlist"]
 
             # Step 2: retrieves the detailed information for each uid
             # returned, in groups of SRA_SEARCH_GROUP_SIZE.
-            if not uids:
+            if not self.uids:
                 print(
                     f"No results found for the following search query: \n {self.fields}"
                 )
                 return  # If no queries found, return nothing
-            pbar = tqdm(total=len(uids))
-            for i in range(0, len(uids), SRA_SEARCH_GROUP_SIZE):
+            pbar = tqdm(total=len(self.uids))
+            for i in range(0, len(self.uids), SRA_SEARCH_GROUP_SIZE):
                 current_uids = ",".join(
-                    uids[i : min(i + SRA_SEARCH_GROUP_SIZE, len(uids))]
+                    self.uids[i : min(i + SRA_SEARCH_GROUP_SIZE, len(self.uids))]
                 )
-                pbar.update(min(SRA_SEARCH_GROUP_SIZE, len(uids) - i))
+                pbar.update(min(SRA_SEARCH_GROUP_SIZE, len(self.uids) - i))
                 payload2 = {"db": "sra", "retmode": "xml", "id": current_uids}
 
                 r = requests_3_retries().get(
@@ -772,8 +794,7 @@ class SraSearch(QuerySearch):
                 self._format_response(r.raw)
             pbar.close()
             self._format_result()
-            if self.verbosity >= 2:
-                self.df["pmid"] = list(uids)
+
         except requests.exceptions.Timeout:
             sys.exit(f"Connection to the server has timed out. Please retry.")
         except requests.exceptions.HTTPError:
@@ -782,6 +803,14 @@ class SraSearch(QuerySearch):
                 f"\nURL queried: {r.url} \nUser query: {self.fields}"
             )
 
+    def get_uids(self):
+        """Get NCBI uids retrieved during this search query.
+
+        Note: There is a chance that some uids retrieved do not appear in
+        the search result output (Refer to #88)
+        """
+        return self.uids
+
     def _format_query_string(self):
         term = ""
         if self.fields["query"]:
@@ -1195,6 +1224,15 @@ class EnaSearch(QuerySearch):
         sends the user query via requests to ENA API and stores search
         result as an instance attribute in the form of a pandas dataframe
 
+    show_result_statistics()
+        Shows summary information about search results.
+
+    visualise_results()
+        Generate graphs that visualise the search results.
+
+    get_plot_objects():
+        Get the plot objects for plots generated.
+
     _format_query_string()
         formats the input user query into a string
 
@@ -1467,6 +1505,15 @@ class GeoSearch(SraSearch):
         depending on the search query. If query is sent to both APIs,
         the intersection of the two sets of query results are returned.
 
+    show_result_statistics()
+        Shows summary information about search results.
+
+    visualise_results()
+        Generate graphs that visualise the search results.
+
+    get_plot_objects():
+        Get the plot objects for plots generated.
+
     _format_geo_query_string()
         formats the GEO DataSets portion of the input user query into a
         string.
diff --git a/pysradb/utils.py b/pysradb/utils.py
index a7be6d9..53c986d 100644
--- a/pysradb/utils.py
+++ b/pysradb/utils.py
@@ -45,14 +45,13 @@ def requests_3_retries():
     -------
     session: requests.Session
         requests session object that allows 3 retries for server-side
-        errors, for GET and POST requests.
+        errors.
     """
     session = requests.Session()
     retry = Retry(
         total=3,
         backoff_factor=0.5,
         status_forcelist=[500, 502, 503, 504],
-        allowed_methods=["POST", "GET", "HEAD"],
     )
     adapter = HTTPAdapter(max_retries=retry)
     session.mount("http://", adapter)


Test patch:
diff --git a/tests/data/test_search/sra_uids.txt b/tests/data/test_search/sra_uids.txt
new file mode 100644
index 0000000..5a75d5d
--- /dev/null
+++ b/tests/data/test_search/sra_uids.txt
@@ -0,0 +1,49 @@
+246598
+246597
+246596
+246595
+181685
+181684
+181683
+155791
+155790
+130901
+130900
+130899
+130898
+130897
+130896
+130895
+130894
+130893
+130892
+130891
+130890
+130889
+130888
+130887
+130886
+130885
+130884
+130883
+130882
+130881
+130880
+214061
+214060
+214059
+214028
+214027
+213982
+202173
+202172
+202171
+202169
+202168
+202167
+201949
+201948
+201814
+201806
+201805
+201804
diff --git a/tests/test_search.py b/tests/test_search.py
index 9fde418..4703bea 100644
--- a/tests/test_search.py
+++ b/tests/test_search.py
@@ -839,6 +839,13 @@ def sra_formatted_responses_2():
     ]
 
 
+@pytest.fixture(scope="module")
+def sra_uids():
+    with open("./tests/data/test_search/sra_uids.txt", "r") as f:
+        uids = f.read().splitlines()
+    return uids
+
+
 @pytest.fixture(scope="module")
 def ena_responses_json():
     data = []
@@ -920,6 +927,14 @@ def test_sra_search_1():
     assert found_accessions == set(expected_accessions)
 
 
+def test_sra_uids(sra_uids):
+    instance = SraSearch(
+        3, 1000, query="ribosome profiling", publication_date="01-10-2012:01-01-2013"
+    )
+    instance.search()
+    assert instance.get_uids() == sra_uids
+
+
 def test_valid_search_query_1_sra(valid_search_inputs_1):
     expected_query = [
         "covid-19",