Repository: ASFHyP3/hyp3-sdk (language: python)

Issue:
Validation Error in asf_search.get_nearest_neighbors
**Describe the bug**
On running `asf_search.get_nearest_neighbors(reference, max_neighbors=2)`, I am getting `Validation Error: granule_list may not be used in conjunction with other search parameters`.

**To Reproduce**
Steps to reproduce the behavior:
```python
import hyp3_sdk as sdk
from hyp3_sdk import asf_search
from tqdm.auto import tqdm

hyp3 = sdk.HyP3()

granules = [
    'S1A_IW_SLC__1SDV_20180503T005155_20180503T005223_021733_0257F4_665D'
]  

insar_jobs = sdk.Batch()
for reference in tqdm(granules):
    neighbors_metadata = asf_search.get_nearest_neighbors(reference, max_neighbors=2) # <- this line causing error.
    print(neighbors_metadata)
    for secondary_metadata in neighbors_metadata:
        insar_jobs += hyp3.submit_insar_job(reference, secondary_metadata['granuleName'], name='insar-example')
print(insar_jobs)
```
**Error Message**

```python
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/hyp3_sdk/exceptions.py in _raise_for_search_status(response)
     37     try:
---> 38         response.raise_for_status()
     39     except HTTPError:

~/.local/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
    942         if http_error_msg:
--> 943             raise HTTPError(http_error_msg, response=self)
    944 

HTTPError: 400 Client Error: BAD REQUEST for url: https://api.daac.asf.alaska.edu/services/search/param?output=json&platform=S1&granule_list=S1A_IW_SLC__1SDV_20180503T005155_20180503T005223_021733_0257F4_665D

During handling of the above exception, another exception occurred:

ASFSearchError                            Traceback (most recent call last)
<ipython-input-4-decdaac30f9b> in <module>
      3 insar_jobs = sdk.Batch()
      4 for reference in tqdm(granules):
----> 5     neighbors_metadata = asf_search.get_nearest_neighbors(reference, max_neighbors=2)
      6     print(neighbors_metadata)
      7 #     for secondary_metadata in neighbors_metadata:

~/.local/lib/python3.8/site-packages/hyp3_sdk/asf_search.py in get_nearest_neighbors(granule, max_neighbors)
     59     }
     60     response = requests.post(_SEARCH_API, params=params)
---> 61     _raise_for_search_status(response)
     62     references = [r for r in response.json()[0] if not r['processingLevel'].startswith('METADATA_')]
     63     if not references:

~/.local/lib/python3.8/site-packages/hyp3_sdk/exceptions.py in _raise_for_search_status(response)
     39     except HTTPError:
     40         if 400 <= response.status_code < 500:
---> 41             raise ASFSearchError(f'{response} {response.json()["error"]["report"]}')
     42         raise ServerError

ASFSearchError: <Response [400]> Validation Error: granule_list may not be used in conjunction with other search parameters
```

**Expected behavior**
Progress bar with submitted job metadata something like this `2 HyP3 Jobs: 0 succeeded, 0 failed, 0 running, 2 pending.`

**Additional context**
1. https://nbviewer.org/github/ASFHyP3/hyp3-sdk/blob/main/docs/sdk_example.ipynb
2. hyp3_sdk version - 1.4.1
3. python3 version - 3.8.10
4. As a way around to this I copied the source code of `get_nearest_neighbors` , disabled the extra search params such as 
 `'platform': 'S1'` and it worked. 



Discussion / hints:
@saheelBreezo  I have the same issue, however, I didn't understand exactly where/how you disabled the params, could you post the code?
Thank you very much
Sure, so the error comes from this line
```python
neighbors_metadata = asf_search.get_nearest_neighbors(reference, max_neighbors=2) 
```
which basically searches the best possible candidate for an InSAR pair given a reference. Now the error says  `granule_list may not be used in conjunction with other search parameters`. So I pulled the src code for this function and disabled the `platform` parameter and it worked.

```python
import requests
from tqdm.auto import tqdm
from hyp3_sdk.exceptions import ASFSearchError, _raise_for_search_status

_SEARCH_API = 'https://api.daac.asf.alaska.edu/services/search/param'

def _get_matching_polarizations(input_polarization: str):
    if input_polarization in ('VV', 'VV+VH'):
        return 'VV,VV+VH'
    if input_polarization in ('HH', 'HH+HV'):
        return 'HH,HH+HV'
    return 

def get_nearest_neighbors(granule: str, max_neighbors: int = 2,):
    """Get a Sentinel-1 granule's nearest neighbors from a temporal stack (backwards in time)

    Args:
        granule: reference granule
        max_neighbors: maximum number of neighbors to return

    Returns:
        neighbors: a list of neighbors sorted by time
    """
    params = {
        'output': 'json',  # jsonlite doesn't include centerLat/centerLon
#         'platform': 'S1',
        'granule_list': granule,
    }
    response = requests.post(_SEARCH_API, params=params)
    _raise_for_search_status(response)
    references = [r for r in response.json()[0] if not r['processingLevel'].startswith('METADATA_')]
    if not references:
        raise ASFSearchError(f'Reference Sentinel-1 granule {granule} could not be found')
    reference = references[0]

    params = {
        'output': 'jsonlite',
        'platform': 'S1',
        'intersectsWith': f'POINT ({reference["centerLon"]} {reference["centerLat"]})',
        'end': reference['startTime'],  # includes reference scene
        'beamMode': reference['beamMode'],
        'flightDirection': reference['flightDirection'],
        'processingLevel': reference['processingLevel'],
        'relativeOrbit': reference['relativeOrbit'],
        'polarization': _get_matching_polarizations(reference['polarization']),
        'lookDirection': reference['lookDirection'],
    }
    response = requests.post(_SEARCH_API, params=params)
    _raise_for_search_status(response)
    neighbors = sorted(response.json()['results'], key=lambda x: x['startTime'], reverse=True)
    return neighbors[1:max_neighbors+1]

granules = [
    'S1A_IW_SLC__1SDV_20180503T005155_20180503T005223_XXXXX_XXXXX_XXX'
]

insar_jobs = sdk.Batch()
for reference in tqdm(granules):
    neighbors_metadata = get_nearest_neighbors(reference, max_neighbors=2)
    for secondary_metadata in neighbors_metadata:
        print(secondary_metadata)
        insar_jobs += hyp3.submit_insar_job(reference, secondary_metadata['granuleName'], name='insar-example')
print(insar_jobs)
```
Having said that this is not a solution but just a hack or workaround for this particular error. Also, if you already have the master and slave id's you can directly send to `hyp3.submit_insar_job` skipping the `nearest neighbour` search.

### Reference
- https://github.com/ASFHyP3/hyp3-sdk/blob/develop/hyp3_sdk/asf_search.py
Awesome, its working! Thank you so much!
Thanks for the report @saheelBreezo and @anaptoro! Unfortunately, the search API recently had to be changed to not accept additional search parameters when doing a list search, so this function is currently broken.

We've depreciated everything in `hyp3_sdk.asf_search` since [v1.4.0](https://github.com/ASFHyP3/hyp3-sdk/releases/tag/v1.4.0) (Sept.) in favor or the much more comprehensive `asf_search` Python package:
* GitHub: https://github.com/asfadmin/Discovery-asf_search
* PyPI: https://pypi.org/project/asf-search/
* Conda-forge: https://anaconda.org/conda-forge/asf_search

which means we are more likely to remove `hyp3_sdk.asf_search` in the next release than fixing this. To get nearest neighbors from the `asf_search` Python package, just like our `hyp3_sdk.asf_search.get_nearest_neighbors` function, you can do something like:

```python
from typing import Optional, Union

import asf_search as asf
import hyp3_sdk as sdk
from tqdm.auto import tqdm


def get_nearest_neighbors(granule: Union[str, asf.ASFProduct], max_neighbors: Optional[int] = 2) -> asf.ASFSearchResults:
    if isinstance(granule, str):
        granule = asf.granule_search(granule)[-1]
    stack = reversed([item for item in granule.stack() if item.properties['temporalBaseline'] < 0])
    return asf.ASFSearchResults(stack)[:max_neighbors]


hyp3 = sdk.HyP3()

granules = [
    'S1A_IW_SLC__1SDV_20180503T005155_20180503T005223_021733_0257F4_665D',
]


insar_jobs = sdk.Batch()
for reference in tqdm(granules):
    neighbors = get_nearest_neighbors(reference, max_neighbors=2)
    for secondary in neighbors:
        insar_jobs += hyp3.submit_insar_job(reference, secondary.properties['sceneName'], name='insar-example')

print(insar_jobs)
```

(cc @glshort)
Thanks @jhkennedy for the clarification . We can close this issue.
@saheelBreezo I'll leave it open until we fix our documentation; it's definitely broken. And thank you for reporting it and posting a solution!

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 55972fc..cf9b3d8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,13 @@ and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [1.5.0]
+
+### Removed
+- `hyp3_sdk.asf_search` has been removed and its functionality has been superseded by the
+  [`asf_search`](https://github.com/asfadmin/Discovery-asf_search) Python package, which provides a more comprehensive
+  ASF search experience and is available on conda-forge and PyPI.
+
 ## [1.4.1](https://github.com/ASFHyP3/hyp3-sdk/compare/v1.4.1...v1.4.1)
 
 ### Fixed
diff --git a/docs/sdk_example.ipynb b/docs/sdk_example.ipynb
index f407728..2c99ad8 100644
--- a/docs/sdk_example.ipynb
+++ b/docs/sdk_example.ipynb
@@ -26,7 +26,25 @@
     "python -m pip install hyp3_sdk\n",
     "```\n",
     "\n",
-    "Full documentation of the SDK can be found in the [HyP3 documentation](https://hyp3-docs.asf.alaska.edu/using/sdk/)"
+    "Full documentation of the SDK can be found in the [HyP3 documentation](https://hyp3-docs.asf.alaska.edu/using/sdk/).\n",
+    "\n",
+    "### Other recommended packages\n",
+    "\n",
+    "We also recommend installing the `asf_search` Python package for performing searches of the ASF catalog. The ASF Search\n",
+    "Python package can be installed using [Anaconda/Miniconda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda)\n",
+    "(recommended) via [`conda`](https://anaconda.org/conda-forge/asf_search):\n",
+    "\n",
+    "```\n",
+    "conda install -c conda-forge asf_search\n",
+    "```\n",
+    "\n",
+    "Or using [`pip`](https://pypi.org/project/asf-search):\n",
+    "\n",
+    "```\n",
+    "python -m pip install asf_search\n",
+    "```\n",
+    "\n",
+    "Full documentation of `asf_search` can be found in the [ASF search documentation](https://docs.asf.alaska.edu/asf_search/basics)."
    ]
   },
   {
@@ -36,8 +54,8 @@
    "outputs": [],
    "source": [
     "# initial setup\n",
-    "import hyp3_sdk as sdk\n",
-    "from hyp3_sdk import asf_search"
+    "import asf_search as asf\n",
+    "import hyp3_sdk as sdk"
    ]
   },
   {
@@ -128,11 +146,46 @@
    "source": [
     "### Submitting Sentinel-1 InSAR jobs\n",
     "\n",
-    "The SDK can also submit Sentinel-1 Interferometric Synthetic Aperture Radar (InSAR) jobs. Using the example granule list\n",
-    "for our RTC jobs as the reference scenes, we can find their nearest and next-nearest neighbor granules, and submit them\n",
-    "as pairs for InSAR processing.\n"
+    "The SDK can also submit Sentinel-1 Interferometric Synthetic Aperture Radar (InSAR) jobs which processes\n",
+    "reference and secondary granule pairs.\n",
+    "\n",
+    "For a particular reference granule, we may want to use the nearest and next-nearest temporal neighbor granules as secondary\n",
+    "scenes. To programmatically find our secondary granules for a reference granule, We'll define a `get_nearest_neighbors`\n",
+    "function that uses the [baseline stack](https://docs.asf.alaska.edu/asf_search/ASFProduct/#stack) method from `asf_search`:"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from typing import Optional\n",
+    "\n",
+    "def get_nearest_neighbors(granule: str, max_neighbors: Optional[int] = None) -> asf.ASFSearchResults:\n",
+    "    granule = asf.granule_search(granule)[-1]\n",
+    "    stack = reversed([item for item in granule.stack() if item.properties['temporalBaseline'] < 0])\n",
+    "    return asf.ASFSearchResults(stack)[:max_neighbors]"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Now, using the example granule list for our RTC jobs as the reference scenes, we can find their nearest and next-nearest neighbor granules, and submit them\n",
+    "as pairs for InSAR processing."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -143,9 +196,9 @@
     "\n",
     "insar_jobs = sdk.Batch()\n",
     "for reference in tqdm(granules):\n",
-    "    neighbors_metadata = asf_search.get_nearest_neighbors(reference, max_neighbors=2)\n",
-    "    for secondary_metadata in neighbors_metadata:\n",
-    "        insar_jobs += hyp3.submit_insar_job(reference, secondary_metadata['granuleName'], name='insar-example')\n",
+    "    neighbors = get_nearest_neighbors(reference, max_neighbors=2)\n",
+    "    for secondary in neighbors:\n",
+    "        insar_jobs += hyp3.submit_insar_job(reference, secondary.properties['sceneName'], name='insar-example')\n",
     "print(insar_jobs)"
    ]
   },
@@ -332,9 +385,13 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   },
    "source": [
-    "*Note: only succeeded jobs will have files to download.*"
+    "*Note: only succeeded jobs will have files to download.*\n"
    ]
   }
  ],
diff --git a/hyp3_sdk/asf_search.py b/hyp3_sdk/asf_search.py
deleted file mode 100644
index eb52408..0000000
--- a/hyp3_sdk/asf_search.py
+++ /dev/null
@@ -1,90 +0,0 @@
-import warnings
-from typing import Iterable, List, Union
-
-import requests
-
-from hyp3_sdk.exceptions import ASFSearchError, _raise_for_search_status
-
-warnings.warn('\nhyp3_sdk.asf_search is deprecated and functionality has been '
-              'superseded by the `asf_search` package available on conda-forge and PyPI. '
-              'See: https://github.com/asfadmin/Discovery-asf_search', FutureWarning)
-
-_SEARCH_API = 'https://api.daac.asf.alaska.edu/services/search/param'
-
-
-def get_metadata(granules: Union[str, Iterable[str]]) -> Union[dict, List[dict]]:
-    """Get the metadata for a granule or list of granules
-
-    Args:
-        granules: granule(s) to lookup metadata for
-
-    Returns:
-        metadata: metadata for the granule(s)
-    """
-    if isinstance(granules, str):
-        granule_list = granules
-    else:
-        granule_list = ','.join(granules)
-
-    params = {
-        'output': 'jsonlite',
-        'granule_list': granule_list,
-    }
-    response = requests.post(_SEARCH_API, params=params)
-    _raise_for_search_status(response)
-
-    metadata = [result for result in response.json()['results']
-                if not result['productType'].startswith('METADATA_')]
-
-    if isinstance(granules, str):
-        return metadata[0]
-
-    return metadata
-
-
-def get_nearest_neighbors(granule: str, max_neighbors: int = 2,) -> List[dict]:
-    """Get a Sentinel-1 granule's nearest neighbors from a temporal stack (backwards in time)
-
-    Args:
-        granule: reference granule
-        max_neighbors: maximum number of neighbors to return
-
-    Returns:
-        neighbors: a list of neighbors sorted by time
-    """
-    params = {
-        'output': 'json',  # jsonlite doesn't include centerLat/centerLon
-        'platform': 'S1',
-        'granule_list': granule,
-    }
-    response = requests.post(_SEARCH_API, params=params)
-    _raise_for_search_status(response)
-    references = [r for r in response.json()[0] if not r['processingLevel'].startswith('METADATA_')]
-    if not references:
-        raise ASFSearchError(f'Reference Sentinel-1 granule {granule} could not be found')
-    reference = references[0]
-
-    params = {
-        'output': 'jsonlite',
-        'platform': 'S1',
-        'intersectsWith': f'POINT ({reference["centerLon"]} {reference["centerLat"]})',
-        'end': reference['startTime'],  # includes reference scene
-        'beamMode': reference['beamMode'],
-        'flightDirection': reference['flightDirection'],
-        'processingLevel': reference['processingLevel'],
-        'relativeOrbit': reference['relativeOrbit'],
-        'polarization': _get_matching_polarizations(reference['polarization']),
-        'lookDirection': reference['lookDirection'],
-    }
-    response = requests.post(_SEARCH_API, params=params)
-    _raise_for_search_status(response)
-    neighbors = sorted(response.json()['results'], key=lambda x: x['startTime'], reverse=True)
-    return neighbors[1:max_neighbors+1]
-
-
-def _get_matching_polarizations(input_polarization: str):
-    if input_polarization in ('VV', 'VV+VH'):
-        return 'VV,VV+VH'
-    if input_polarization in ('HH', 'HH+HV'):
-        return 'HH,HH+HV'
-    return input_polarization


Test patch:
diff --git a/tests/test_asf_search.py b/tests/test_asf_search.py
deleted file mode 100644
index 3afa0f0..0000000
--- a/tests/test_asf_search.py
+++ /dev/null
@@ -1,121 +0,0 @@
-import pytest
-import responses
-
-from hyp3_sdk import asf_search
-from hyp3_sdk.exceptions import ASFSearchError
-
-
-@responses.activate
-def test_get_metadata_string():
-    responses.add(responses.POST, asf_search._SEARCH_API,
-                  json={'results': [
-                      {'productType': 'METADATA_SLC'},
-                      {'productType': 'SLC'},
-                  ]})
-    assert asf_search.get_metadata('S1B_IW_SLC_') == {'productType': 'SLC'}
-
-
-@responses.activate
-def test_get_metadata_list():
-    mock_response = {
-        'results': [
-            {'productType': 'GRD_HD'},
-            {'productType': 'METADATA_GRD'},
-            {'productType': 'METADATA_SLC'},
-            {'productType': 'SLC'},
-        ]
-    }
-    responses.add(responses.POST, asf_search._SEARCH_API, json=mock_response)
-
-    assert asf_search.get_metadata(['S1B_IW_SLC_', 'S1B_IW_SLC_']) == \
-           [{'productType': 'GRD_HD'}, {'productType': 'SLC'}]
-
-
-@responses.activate
-def test_get_nearest_neighbor():
-    reference = 'S1A_IW_SLC__1SDV_20161106T002750_20161106T002817_013814_016338_19F7'
-
-    reference_query = f'{asf_search._SEARCH_API}' \
-                      '?output=json' \
-                      '&platform=S1' \
-                      f'&granule_list={reference}'
-    reference_response = [[
-        {
-            'centerLon': '0.0',
-            'centerLat': '1.0',
-            'startTime': '2016-11-06T00:27:50.000000',
-            'beamMode': 'IW',
-            'flightDirection': 'DESCENDING',
-            'processingLevel': 'SLC',
-            'relativeOrbit': '1',
-            'polarization': 'VV+VH',
-            'lookDirection': 'R',
-        },
-        {
-            'centerLon': 'x',
-            'centerLat': 'x',
-            'startTime': 'x',
-            'beamMode': 'x',
-            'flightDirection': 'x',
-            'processingLevel': 'METADATA_SLC',
-            'relativeOrbit': 'x',
-            'polarization': 'x',
-            'lookDirection': 'x',
-        },
-    ]]
-    responses.add(responses.POST, reference_query, json=reference_response)
-
-    stack_query = f'{asf_search._SEARCH_API}' \
-                  '?output=jsonlite' \
-                  '&platform=S1' \
-                  '&intersectsWith=POINT+(0.0+1.0)' \
-                  '&end=2016-11-06T00:27:50.000000' \
-                  '&beamMode=IW' \
-                  '&flightDirection=DESCENDING' \
-                  '&processingLevel=SLC' \
-                  '&relativeOrbit=1' \
-                  '&polarization=VV%2CVV%2BVH' \
-                  '&lookDirection=R'
-    stack_response = {
-        'results': [
-            {'startTime': 1},
-            {'startTime': 5},
-            {'startTime': 4},
-            {'startTime': 2},
-            {'startTime': 3},
-        ]
-    }
-    responses.add(responses.POST, stack_query, json=stack_response)
-
-    assert asf_search.get_nearest_neighbors(reference) == \
-           [{'startTime': 4}, {'startTime': 3}]
-    assert asf_search.get_nearest_neighbors(reference, max_neighbors=3) == \
-           [{'startTime': 4}, {'startTime': 3}, {'startTime': 2}]
-
-
-@responses.activate
-def test_get_nearest_neighbors_no_reference():
-    responses.add(responses.POST, asf_search._SEARCH_API, json=[[]])
-    with pytest.raises(ASFSearchError) as e:
-        asf_search.get_nearest_neighbors('foo')
-    assert 'foo' in str(e)
-
-
-@responses.activate
-def test_get_nearest_neighbors_no_reference_only_metadata():
-    responses.add(responses.POST, asf_search._SEARCH_API, json=[[{'processingLevel': 'METADATA_SLC'}]])
-    with pytest.raises(ASFSearchError) as e:
-        asf_search.get_nearest_neighbors('foo')
-    assert 'foo' in str(e)
-
-
-def test_get_matching_polarizations():
-    assert asf_search._get_matching_polarizations('VV') == 'VV,VV+VH'
-    assert asf_search._get_matching_polarizations('VV+VH') == 'VV,VV+VH'
-    assert asf_search._get_matching_polarizations('HH') == 'HH,HH+HV'
-    assert asf_search._get_matching_polarizations('HH+HV') == 'HH,HH+HV'
-    assert asf_search._get_matching_polarizations('HH') == 'HH,HH+HV'
-    assert asf_search._get_matching_polarizations('Dual VV') == 'Dual VV'
-    assert asf_search._get_matching_polarizations('Dual VH') == 'Dual VH'
-    assert asf_search._get_matching_polarizations('Dual HH') == 'Dual HH'
-    assert asf_search._get_matching_polarizations('Dual HV') == 'Dual HV'