Repository: astropy/pyvo
Pull request #218 (2020-02-27 04:51:13)

Batched DataLink calls
#217

andamian: First attempt. I've tried to make it a drop in change, transparent change. A bit tricky since `DatalinkResults` was designed to hold entries for a single ID. The work around was to cache multiple calls to datalink but return only rows that belong to a specific ID. Not that clean, I admit but it seems to work. I kept the old `DatalinkResultsMixin.iter_datalinks` method for now just a bench mark for correctness and performance but it can be replaced by the corresponding `iter_datalinks` without any effect (hopefully) on the calling code. 
@msdemlei & @funbaker or anyone else, please let me know if you can suggest a better approach. I wouldn't leave the decision on the batch size on the user, but the variable can be tweaked. This is the first version so it's not entirely optimized but it works with a few real examples I've tried.
msdemlei: On Wed, Feb 26, 2020 at 08:59:36PM -0800, andamian wrote:
> @msdemlei & @funbaker or anyone else, please let me know if you can
> suggest a better approach. I wouldn't leave the decision on the
> batch size on the user, but the variable can be tweaked. This is
> the first version so it's not entirely optimized but it works with
> a few real examples I've tried.

I suppose given the iter_datalinks approach this is about how it
should be done.  There is a bit of a problem with this approach
because datalink services can have match limits.  10000 for such a
limit is plausible, and 10 datalinks per id is also not unheard-of (I
have some datasets with ~100 datalinks).

Hence, for DATALINK_BATCH_SIZE=1000 you might well run into a match
limit.  That, I think, shouldn't pass unnoticed.  The way the spec is
written, it's relatively simple to catch such a situation: After
doing the batched query, you could do the equivalent (I've not lookad
at what sort of thing _get_datalinks actualy returns) of

```
  if set(ids)!=set(r["ID"] for r in self._cached_dl):
    raise WhateverError("Datalink batched query overflowed match limit;"
      " Decreate DATALINK_BATCH_CALL_SIZE.")
```

To make the code a bit more transparent, I'd pull out the cache
filling into a separate method.

You can also pass MAXREC to your datalink call, but since we have no
way of determining the hard limit on the service at this point, I
don't think that's useful until we've thought a bit more about
additional service metadata (which brings us back to caproles...).

       -- Markus

andamian: @msdemlei the batch size is in the input but the server limitation is in the output. The client can't come up with the optimal batch size since it doesn't know how many data links it would generate. Each different call with the same batch size can generate a different number of output rows.

I was thinking of a different approach: send all the IDs at once, and if the server returns OVERFLOW status remove the processed ones from the list before sending them again. Continue until status OK. Do you think this will put unnecessary burden on the service?
To accommodate large number of IDs we might need to change the submission from the current GET to a POST but I think the spec supports that.

What do you think?


andamian: BTW, one of the scenarios that astronomers have asked is:
1. Perform SIA2 query
2. Process results which typically is just some further filtering that cannot be achieved with the query.
3. Download data
How can step 2 be performed with the `SIAResults` object? The modified `SIAResults` object is required in step 3 since it stores information about the DataLink step. Users should be able to easily perform record deletes from an `SIAResults`, no? Maybe the functionality is there but I'm missing it.
msdemlei: On Thu, Feb 27, 2020 at 11:45:51AM -0800, andamian wrote:
> I was thinking of a different approach: send all the IDs at once,
> and if the server returns OVERFLOW status remove the processed ones
> from the list before sending them again. Continue until status OK.
> Do you think this will put unnecessary burden on the service?

I think that's a totally reasonable thing to do.  If you're going to
iter_datalinks fairly exhaustively anyway, the server will certainly
be grateful if you're firing off a few large requests rather than a
couple of thousand small requests.

Perhaps one needs to think about really large queries, as in several
1e5 or so; a datalink record with an attached service block may be a
few k, so the results may work out to be in the Gigabyte range.
But then that should probably be done on the server side by
introducing a reasonable match limit there.

> To accommodate large number of IDs we might need to change the
> submission from the current GET to a POST but I think the spec
> supports that.

It should definitely be POST either way; Datalink sect. 2.1 requires
support for both, and the only reason I see for having GET in the
first place is so people can pass around canned queries as URL
literals.  For dynamic queries, POST is a lot more robust all around.

codecov[bot]: # [Codecov](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=h1) Report
> Merging [#218](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=desc) into [master](https://codecov.io/gh/astropy/pyvo/commit/2c59cea8213d7db769ca8f62385438d83427b822?src=pr&el=desc) will **decrease** coverage by `0.5%`.
> The diff coverage is `15.38%`.

[![Impacted file tree graph](https://codecov.io/gh/astropy/pyvo/pull/218/graphs/tree.svg?width=650&token=Mynyo9xoPZ&height=150&src=pr)](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #218      +/-   ##
==========================================
- Coverage   72.08%   71.57%   -0.51%     
==========================================
  Files          40       40              
  Lines        4402     4440      +38     
==========================================
+ Hits         3173     3178       +5     
- Misses       1229     1262      +33
```


| [Impacted Files](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pyvo/dal/adhoc.py](https://codecov.io/gh/astropy/pyvo/pull/218/diff?src=pr&el=tree#diff-cHl2by9kYWwvYWRob2MucHk=) | `54.13% <15.38%> (-4.98%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=footer). Last update [2c59cea...21ea3a2](https://codecov.io/gh/astropy/pyvo/pull/218?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

andamian: @msdemlei @funbaker I've pushed another version. Initially, the client sends all the IDs with a POST. If the result is incomplete, the size of the returned result represents the size of the batch. Subsequent calls are initiated until all the IDs are resolved.
Please review at your earliest convenience. Thanks
andamian: Fixed tests broken due to upstream changes (https://github.com/astropy/astropy/pull/9505 )

trjaffe: This doesn't work for the HEASARC TAP service's DataLinks.  The reason is that our service uses the integer row number as the ID.  So in the VOTable, there's:

>    \<FIELD ID="DataLinkID" datatype="int" name="__row">
> ...
>  \<RESOURCE type="meta" utype="adhoc:service">
>   \<PARAM ID="standardID" arraysize="*" datatype="char" name="standardID" value="ivo://ivoa.net/std/DataLink#links-1.0"/>
>   \<PARAM ID="accessURL" arraysize="*" datatype="char" name="accessURL" value="https://heasarc.gsfc.nasa.gov/xamin/vo/datalink/chanmaster"/>
>  \</RESOURCE>


Then in adhoc.py:213, the following code

`                    self._current_ids = list(OrderedDict.fromkeys(
                        [_ for _ in self._current_batch.to_table()['ID']]))
`
gets the IDs from the DataLinkResults as a list of strings, while the self._remaining_ids are from the TAPResults votable as integers.  So then the remove() further down doesn't work.  

So Tom McGlynn sent a message to the DAL group asking about this, since the standard document doesn't make clear that IDs have to be strings.  So which needs to be fixed, our TAP service to use datatype="char" for the row or PyVO to convert in the above check?  I found two places that a simple type conversion would fix this in PyVO.  (The other is adhoc.py:556 for the same reason.)  Of course, it could also be fixed in our service.  

Thoughts?

funbaker: summoning @msdemlei for a standard related question
msdemlei: On Wed, Apr 01, 2020 at 10:30:18AM -0700, trjaffe wrote:
> So Tom McGlynn sent a message to the DAL group asking about this,
> since the standard document doesn't make clear that IDs have to be
> strings.  So which needs to be fixed, our TAP service to use
> datatype="char" for the row or PyVO to convert in the above check?
> I found two places that a simple type conversion would fix this in
> PyVO.  (The other is adhoc.py:556 for the same reason.)  Of course,
> it could also be fixed in our service.  
> 
> Thoughts?

I'm not so sure about the "simple type conversion" -- type
conversions have a way of eventually turning out hairy.

Anyway, this is clearly a lacuna in the specs; actually, both VOTable
(which is hardly to blame) and Datalink, so I don't think we can
simply decide this here.  My current position is in

http://mail.ivoa.net/pipermail/dal/2020-April/008322.html

-- were I don't think any amount of discussion will change my stance
on avoiding different type/arraysize/xtype tuples on the two ends of
a ./@ref relationship.

funbaker: I see two possibilities:

1. make everything `char`
2. do implicit conversion but raise a warning if it's with non-trivial types in terms of conversion.
andamian: > A belated update: the DAL group is going to clarify the standard that the ID should be char. We haven't yet finished fixing our service, but this mod works with others and looks fine to everybody else. Nor did I have any other comments. So I think this PR is good to go.

Thank you @trjaffe for looking into it.
andamian: @funbaker - I think I've now addressed all the concerns. Please let me know if that's not the case. Thanks
funbaker: > @funbaker - I think I've now addressed all the concerns. Please let me know if that's not the case. Thanks

Aye, should be fine for now.

Patch:
diff --git a/pyvo/dal/adhoc.py b/pyvo/dal/adhoc.py
index 5d0ecf7..388c690 100644
--- a/pyvo/dal/adhoc.py
+++ b/pyvo/dal/adhoc.py
@@ -4,6 +4,8 @@ Datalink classes and mixins
 """
 import numpy as np
 import warnings
+import copy
+from collections import OrderedDict
 
 from .query import DALResults, DALQuery, DALService, Record
 from .exceptions import DALServiceError
@@ -14,6 +16,7 @@ from astropy.io.votable.tree import Param
 from astropy import units as u
 from astropy.units import Quantity, Unit
 from astropy.units import spectral as spectral_equivalencies
+from pyvo.utils.compat import ASTROPY_LT_4_1
 
 from astropy.io.votable.tree import Resource, Group
 from astropy.utils.collections import HomogeneousList
@@ -23,6 +26,16 @@ from .params import PosQueryParam, IntervalQueryParam, TimeQueryParam,\
     EnumQueryParam
 from ..dam.obscore import POLARIZATION_STATES
 
+# calls to DataLink from the results pages are batched for performance
+# reasons. This is the size of a batch
+DATALINK_BATCH_CALL_SIZE = 50
+
+SODA_SYNC_IVOID = 'ivo://ivoa.net/std/SODA#sync-1.0'
+DATALINK_IVOID = 'ivo://ivoa.net/std/datalink'
+
+# MIME types
+DATALINK_MIME_TYPE = 'application/x-votable+xml;content=datalink'
+
 
 # monkeypatch astropy with group support in RESOURCE
 def _monkeypath_astropy_resource_groups():
@@ -118,6 +131,10 @@ class AdhocServiceResultsMixin:
         Resource
             The resource element describing the service.
         """
+        if ASTROPY_LT_4_1 and isinstance(ivo_id, str):
+            ivo_id = ivo_id.encode('utf-8')
+        if not ASTROPY_LT_4_1 and isinstance(ivo_id, bytes):
+            ivo_id = ivo_id.decode('utf-8')
         for adhocservice in self.iter_adhocservices():
             if any(
                     all((
@@ -154,12 +171,59 @@ class DatalinkResultsMixin(AdhocServiceResultsMixin):
     """
     Mixing for datalink functionallity for results classes.
     """
+
     def iter_datalinks(self):
         """
         Iterates over all datalinks in a DALResult.
         """
-        for record in self:
-            yield record.getdatalink()
+        # To reduce the number of calls to the Datalink service, multiple
+        # IDs are sent in batches. The appropriate batch size is not available
+        # as each service has its own criteria for determining the maximum
+        # number of IDs processed per request. To overcome this limitation,
+        # this method initially sends all the available IDs and if the
+        # response is partial, uses the size of the response as the batch
+        # size.
+        if not hasattr(self, '_datalink'):
+            try:
+                self._datalink = self.get_adhocservice_by_ivoid(DATALINK_IVOID)
+            except DALServiceError:
+                self._datalink = None
+        remaining_ids = []  # remaining IDs to processed
+        current_batch = None  # retrieved but not returned yet
+        current_ids = []  # retrieved but not returned
+        processed_ids = []  # retrived and returned IDs
+        batch_size = None  # size of the batch
+
+        for row in self:
+            if self._datalink:
+                if not current_ids:
+                    if batch_size is None:
+                        # first call.
+                        self.query = DatalinkQuery.from_resource(
+                            [_ for _ in self], self._datalink)
+                        remaining_ids = self.query['ID']
+                    if not remaining_ids:
+                        # we are done
+                        return
+                    if batch_size:
+                        # subsequent calls are limitted to batch size
+                        self.query['ID'] = remaining_ids[:batch_size]
+                    current_batch = self.query.execute(post=True)
+                    current_ids = list(OrderedDict.fromkeys(
+                        [_ for _ in current_batch.to_table()['ID']]))
+                    if not current_ids:
+                        raise DALServiceError(
+                            'Could not retrieve datalinks for: {}'.format(
+                                ', '.join([_ for _ in remaining_ids])))
+                    batch_size = len(current_ids)
+                id = current_ids.pop(0)
+                processed_ids.append(id)
+                remaining_ids.remove(id)
+                yield current_batch.clone_byid(id)
+            elif row.access_format == DATALINK_MIME_TYPE:
+                yield DatalinkResults.from_result_url(row.getdataurl())
+            else:
+                yield None
 
 
 class DatalinkRecordMixin:
@@ -170,8 +234,7 @@ class DatalinkRecordMixin:
     """
     def getdatalink(self):
         try:
-            datalink = self._results.get_adhocservice_by_ivoid(
-                b"ivo://ivoa.net/std/datalink")
+            datalink = self._results.get_adhocservice_by_ivoid(DATALINK_IVOID)
 
             query = DatalinkQuery.from_resource(self, datalink)
             return query.execute()
@@ -281,9 +344,9 @@ class DatalinkQuery(DALQuery):
     allowing the caller to take greater control of the result processing.
     """
     @classmethod
-    def from_resource(cls, row, resource, session=None, **kwargs):
+    def from_resource(cls, rows, resource, session=None, **kwargs):
         """
-        Creates a instance from a Record and a Datalink Resource.
+        Creates a instance from a number of records and a Datalink Resource.
 
         XML Hierarchy:
 
@@ -309,7 +372,13 @@ class DatalinkQuery(DALQuery):
         query_params = dict()
         for name, input_param in input_params.items():
             if input_param.ref:
-                query_params[name] = row[input_param.ref]
+                if isinstance(rows, list):
+                    query_params[name] = []
+                    for r in rows:
+                        query_params[name].append(r[input_param.ref])
+                else:
+                    # scalars are also accepted for backwards compatibility
+                    query_params[name] = rows[input_param.ref]
             elif np.isscalar(input_param.value) and input_param.value:
                 query_params[name] = input_param.value
             elif (
@@ -323,6 +392,8 @@ class DatalinkQuery(DALQuery):
         for name, query_param in kwargs.items():
             try:
                 input_param = find_param_by_keyword(name, input_params)
+                if input_param and query_param is None:
+                    del query_params[input_param.name]
                 converter = get_converter(input_param)
                 query_params[input_param.name] = converter.serialize(
                     query_param)
@@ -354,7 +425,7 @@ class DatalinkQuery(DALQuery):
         if responseformat:
             self["RESPONSEFORMAT"] = responseformat
 
-    def execute(self):
+    def execute(self, post=False):
         """
         submit the query and return the results as a DatalinkResults instance
 
@@ -368,7 +439,8 @@ class DatalinkQuery(DALQuery):
         DALFormatError
            for errors parsing the VOTable response
         """
-        return DatalinkResults(self.execute_votable(), url=self.queryurl, session=self._session)
+        return DatalinkResults(self.execute_votable(post),
+                               url=self.queryurl, session=self._session)
 
 
 class DatalinkResults(DatalinkResultsMixin, DALResults):
@@ -458,6 +530,41 @@ class DatalinkResults(DatalinkResultsMixin, DALResults):
             if record.semantics == semantics:
                 yield record
 
+    def clone_byid(self, id):
+        """
+        return a clone of the object with results and corresponding
+         resources matching a given id
+
+        Returns
+        -------
+        Sequence of DatalinkRecord
+            a sequence of dictionary-like wrappers containing the result record
+        """
+
+        copy_tb = copy.deepcopy(self.votable)
+        votable = copy_tb.get_first_table()
+        # find index of ID column
+        id_index = None
+        for index, field in enumerate(votable.fields):
+            if field.name == 'ID':
+                id_index = index
+        rows = [x for x in votable.array if x[id_index] == id]
+        votable.create_arrays(len(rows))
+        for index, row in enumerate(rows):
+            votable.array[index] = row
+        # now remove unreferenced services from resources
+        if ASTROPY_LT_4_1:
+            referenced_serviced = \
+                [x.decode('utf-8') for x in votable.array['service_def'] if x]
+        else:
+            referenced_serviced = \
+                [x for x in votable.array['service_def'] if x]
+        # remove customized that are not referenced by the current results
+        for x in copy_tb.resources:
+            if x.ID and x.ID not in referenced_serviced:
+                copy_tb.resources.remove(x)
+        return DatalinkResults(copy_tb)
+
     def getdataset(self, timeout=None):
         """
         return the first row with the dataset identified by semantics #this
@@ -491,21 +598,20 @@ class DatalinkResults(DatalinkResultsMixin, DALResults):
 
 class SodaRecordMixin:
     """
-    Mixin for soda functionallity for record classes.
+    Mixin for soda functionality for record classes.
     If used, it's result class must have
     `pyvo.dal.datalink.AdhocServiceResultsMixin` mixed in.
     """
     def _get_soda_resource(self):
         try:
-            return self._results.get_adhocservice_by_ivoid(
-                b"ivo://ivoa.net/std/SODA#sync")
+            return self._results.get_adhocservice_by_ivoid(SODA_SYNC_IVOID)
         except DALServiceError:
             pass
 
         # let it count as soda resource
         try:
             return self._results.get_adhocservice_by_ivoid(
-                b"ivo://ivoa.net/std/datalink#links")
+                "ivo://ivoa.net/std/datalink#links")
         except DALServiceError:
             pass
 
@@ -526,7 +632,7 @@ class SodaRecordMixin:
             try:
                 datalink_result = DatalinkResults.from_result_url(dataurl)
                 return datalink_result.get_adhocservice_by_ivoid(
-                    b"ivo://ivoa.net/std/SODA#sync")
+                    SODA_SYNC_IVOID)
             except DALServiceError:
                 pass
 
diff --git a/pyvo/dal/params.py b/pyvo/dal/params.py
index 42f0a3f..276f913 100644
--- a/pyvo/dal/params.py
+++ b/pyvo/dal/params.py
@@ -87,7 +87,11 @@ class Converter:
         """
         Serialize for use in DAL Queries
         """
-        return str(value)
+        if isinstance(value, list):
+            # multiple values
+            return [str(_) for _ in value]
+        else:
+            return str(value)
 
 
 class Number(Converter):
diff --git a/pyvo/dal/query.py b/pyvo/dal/query.py
index 0c684cf..b3bc2ed 100644
--- a/pyvo/dal/query.py
+++ b/pyvo/dal/query.py
@@ -181,14 +181,14 @@ class DALQuery(dict):
         return out
 
     @stream_decode_content
-    def execute_stream(self):
+    def execute_stream(self, post=False):
         """
         Submit the query and return the raw response as a file stream.
 
         No exceptions are raised here because non-2xx responses might still
         contain payload. They can be raised later by calling ``raise_if_error``
         """
-        response = self.submit()
+        response = self.submit(post=post)
 
         try:
             response.raise_for_status()
@@ -198,17 +198,22 @@ class DALQuery(dict):
         finally:
             return response.raw
 
-    def submit(self):
+    def submit(self, post=False):
         """
         does the actual request
         """
         url = self.queryurl
         params = {k: v for k, v in self.items()}
 
-        response = self._session.get(url, params=params, stream=True, allow_redirects=True)
+        if post:
+            response = self._session.post(url, data=params, stream=True,
+                                          allow_redirects=True)
+        else:
+            response = self._session.get(url, params=params, stream=True,
+                                         allow_redirects=True)
         return response
 
-    def execute_votable(self):
+    def execute_votable(self, post=False):
         """
         Submit the query and return the results as an AstroPy votable instance.
         As this is the level where qualified error messages are available,
@@ -234,7 +239,7 @@ class DALQuery(dict):
         DALQueryError
         """
         try:
-            return votableparse(self.execute_stream().read)
+            return votableparse(self.execute_stream(post=post).read)
         except Exception as e:
             self.raise_if_error()
             raise DALFormatError(e, self.queryurl)
diff --git a/pyvo/dal/tap.py b/pyvo/dal/tap.py
index 4189b78..e446fe8 100644
--- a/pyvo/dal/tap.py
+++ b/pyvo/dal/tap.py
@@ -876,7 +876,7 @@ class TAPQuery(DALQuery):
     def queryurl(self):
         return '{baseurl}/{mode}'.format(baseurl=self.baseurl, mode=self._mode)
 
-    def execute_stream(self):
+    def execute_stream(self, post=False):
         """
         submit the query and return the raw VOTable XML as a file stream
 
@@ -893,7 +893,7 @@ class TAPQuery(DALQuery):
             raise DALServiceError(
                 "Cannot execute a non-synchronous query. Use submit instead")
 
-        return super().execute_stream()
+        return super().execute_stream(post=post)
 
     def execute(self):
         """
@@ -911,7 +911,7 @@ class TAPQuery(DALQuery):
         """
         return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
 
-    def submit(self):
+    def submit(self, post=False):
         """
         Does the request part of the TAP query.
         This function is separated from response parsing because async queries
diff --git a/pyvo/utils/__init__.py b/pyvo/utils/__init__.py
index e69de29..9fee358 100644
--- a/pyvo/utils/__init__.py
+++ b/pyvo/utils/__init__.py
@@ -0,0 +1,1 @@
+from .compat import *
diff --git a/pyvo/utils/compat.py b/pyvo/utils/compat.py
new file mode 100644
index 0000000..88bad5f
--- /dev/null
+++ b/pyvo/utils/compat.py
@@ -0,0 +1,10 @@
+# Licensed under a 3-clause BSD style license - see LICENSE.rst
+"""
+Placeholder for compatibility constructs
+"""
+from distutils.version import LooseVersion
+import astropy
+
+__all__ = ['ASTROPY_LT_4_1']
+
+ASTROPY_LT_4_1 = LooseVersion(astropy.__version__) < '4.1'


Test patch:
diff --git a/pyvo/dal/tests/data/datalink/cutout1.xml b/pyvo/dal/tests/data/datalink/cutout1.xml
new file mode 100644
index 0000000..554c4b5
--- /dev/null
+++ b/pyvo/dal/tests/data/datalink/cutout1.xml
@@ -0,0 +1,217 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.4">
+  <RESOURCE type="results">
+    <TABLE>
+      <FIELD name="ID" datatype="char" arraysize="*" ucd="meta.id;meta.main" />
+      <FIELD name="access_url" datatype="char" arraysize="*" ucd="meta.ref.url" />
+      <FIELD name="service_def" datatype="char" arraysize="*" ucd="meta.ref" />
+      <FIELD name="error_message" datatype="char" arraysize="*" ucd="meta.code.error" />
+      <FIELD name="semantics" datatype="char" arraysize="*" ucd="meta.code" />
+      <FIELD name="description" datatype="char" arraysize="*" ucd="meta.note" />
+      <FIELD name="content_type" datatype="char" arraysize="*" ucd="meta.code.mime" />
+      <FIELD name="content_length" datatype="long" ucd="phys.size;meta.file" unit="byte" />
+      <FIELD name="readable" datatype="boolean">
+        <DESCRIPTION>the caller is allowed to use this link with the current authenticated identity</DESCRIPTION>
+      </FIELD>
+      <DATA>
+        <TABLEDATA>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54150/cal054150r</TD>
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/MACHO/cal054150r.fits.fz?RUNID=e3h6rssb161oyf7j</TD>
+            <TD />
+            <TD />
+            <TD>#this</TD>
+            <TD>download ad:MACHO/cal054150r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD>18616320</TD>
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54150/cal054150r</TD>
+            <TD />
+            <TD>soda-146cec58-9031-4568-9aeb-380eabb50942</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-sync cutout of ad:MACHO/cal054150r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54150/cal054150r</TD>
+            <TD />
+            <TD>soda-142c15fb-f9ff-4ad0-b35f-7efb07d9bfea</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-async cutout of ad:MACHO/cal054150r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151b</TD>
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/MACHO/cal054151b.fits.fz?RUNID=e3h6rssb161oyf7j</TD>
+            <TD />
+            <TD />
+            <TD>#this</TD>
+            <TD>download ad:MACHO/cal054151b.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD>21692160</TD>
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151b</TD>
+            <TD />
+            <TD>soda-2d68bb35-50e4-459b-bfb2-ac2b5da1a5d1</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-sync cutout of ad:MACHO/cal054151b.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151b</TD>
+            <TD />
+            <TD>soda-fadba418-47ec-4a3d-b8ab-8368f3081082</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-async cutout of ad:MACHO/cal054151b.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+        </TABLEDATA>
+      </DATA>
+    </TABLE>
+    <INFO name="QUERY_STATUS" value="OK" />
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-146cec58-9031-4568-9aeb-380eabb50942" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="32" value="ivo://ivoa.net/std/SODA#sync-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="58" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054150r.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="e3h6rssb161oyf7j" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="88.77362846593161 -67.78431533365645 0.5122773056987322" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="89.70783791485876 -67.41840881752057 89.74786512306278 -68.14322955012662 88.22311503462467 -68.14813181576002 87.81208257298478 -67.77896886699358 87.82899304985233 -67.4197778334441" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="5.800000000000001E-7" />
+          <MAX value="7.8E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.522581" />
+          <MAX value="50527.526053" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-142c15fb-f9ff-4ad0-b35f-7efb07d9bfea" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="33" value="ivo://ivoa.net/std/SODA#async-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="59" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054150r.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="e3h6rssb161oyf7j" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="88.77362846593161 -67.78431533365645 0.5122773056987322" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="89.70783791485876 -67.41840881752057 89.74786512306278 -68.14322955012662 88.22311503462467 -68.14813181576002 87.81208257298478 -67.77896886699358 87.82899304985233 -67.4197778334441" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="5.800000000000001E-7" />
+          <MAX value="7.8E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.522581" />
+          <MAX value="50527.526053" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-2d68bb35-50e4-459b-bfb2-ac2b5da1a5d1" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="32" value="ivo://ivoa.net/std/SODA#sync-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="58" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054151b.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="e3h6rssb161oyf7j" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="79.95903372635773 -70.324541763973 0.5210467418210978" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="81.03835269676311 -69.95434045997953 81.07118698080274 -70.68523150491481 78.84035765138141 -70.68806135194416 78.88155357298983 -69.9513684574919" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="4.800000000000001E-7" />
+          <MAX value="5.800000000000001E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.527002" />
+          <MAX value="50527.530475" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-fadba418-47ec-4a3d-b8ab-8368f3081082" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="33" value="ivo://ivoa.net/std/SODA#async-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="59" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054151b.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="e3h6rssb161oyf7j" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="79.95903372635773 -70.324541763973 0.5210467418210978" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="81.03835269676311 -69.95434045997953 81.07118698080274 -70.68523150491481 78.84035765138141 -70.68806135194416 78.88155357298983 -69.9513684574919" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="4.800000000000001E-7" />
+          <MAX value="5.800000000000001E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.527002" />
+          <MAX value="50527.530475" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+</VOTABLE>
diff --git a/pyvo/dal/tests/data/datalink/cutout2.xml b/pyvo/dal/tests/data/datalink/cutout2.xml
new file mode 100644
index 0000000..1555fc3
--- /dev/null
+++ b/pyvo/dal/tests/data/datalink/cutout2.xml
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.4">
+  <RESOURCE type="results">
+    <TABLE>
+      <FIELD name="ID" datatype="char" arraysize="*" ucd="meta.id;meta.main" />
+      <FIELD name="access_url" datatype="char" arraysize="*" ucd="meta.ref.url" />
+      <FIELD name="service_def" datatype="char" arraysize="*" ucd="meta.ref" />
+      <FIELD name="error_message" datatype="char" arraysize="*" ucd="meta.code.error" />
+      <FIELD name="semantics" datatype="char" arraysize="*" ucd="meta.code" />
+      <FIELD name="description" datatype="char" arraysize="*" ucd="meta.note" />
+      <FIELD name="content_type" datatype="char" arraysize="*" ucd="meta.code.mime" />
+      <FIELD name="content_length" datatype="long" ucd="phys.size;meta.file" unit="byte" />
+      <FIELD name="readable" datatype="boolean">
+        <DESCRIPTION>the caller is allowed to use this link with the current authenticated identity</DESCRIPTION>
+      </FIELD>
+      <DATA>
+        <TABLEDATA>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151r</TD>
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/MACHO/cal054151r.fits.fz?RUNID=of051krube49ud2e</TD>
+            <TD />
+            <TD />
+            <TD>#this</TD>
+            <TD>download ad:MACHO/cal054151r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD>18987840</TD>
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151r</TD>
+            <TD />
+            <TD>soda-40d74211-597b-4aa9-a153-f5ac67b77021</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-sync cutout of ad:MACHO/cal054151r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+          <TR>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151r</TD>
+            <TD />
+            <TD>soda-8e48c445-8f76-4c01-abcf-bdb1c2f7caf5</TD>
+            <TD />
+            <TD>#cutout</TD>
+            <TD>SODA-async cutout of ad:MACHO/cal054151r.fits.fz</TD>
+            <TD>application/fits</TD>
+            <TD />
+            <TD>true</TD>
+          </TR>
+        </TABLEDATA>
+      </DATA>
+    </TABLE>
+    <INFO name="QUERY_STATUS" value="OK" />
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-40d74211-597b-4aa9-a153-f5ac67b77021" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="32" value="ivo://ivoa.net/std/SODA#sync-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="58" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054151r.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="of051krube49ud2e" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="79.96919731538502 -70.32069351443877 0.5124495151081784" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="81.01558843560966 -69.95461093880266 81.06567094624911 -70.67927922356017 79.34988020864029 -70.684376201833 78.89013903241694 -70.31495449099631 78.91029632344201 -69.95565099163903" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="5.800000000000001E-7" />
+          <MAX value="7.8E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.527002" />
+          <MAX value="50527.530475" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+  <RESOURCE type="meta" ID="soda-8e48c445-8f76-4c01-abcf-bdb1c2f7caf5" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="26" value="ivo://cadc.nrc.ca/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="33" value="ivo://ivoa.net/std/SODA#async-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="59" value="https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/async" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="27" ucd="meta.id;meta.dataset" value="ad:MACHO/cal054151r.fits.fz" />
+      <PARAM name="RUNID" datatype="char" arraysize="16" ucd="" value="of051krube49ud2e" />
+      <PARAM name="POS" datatype="char" arraysize="*" ucd="obs.field" value="" />
+      <PARAM name="CIRCLE" datatype="double" arraysize="3" ucd="obs.field" unit="deg" xtype="circle" value="">
+        <VALUES>
+          <MAX value="79.96919731538502 -70.32069351443877 0.5124495151081784" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="POLYGON" datatype="double" arraysize="*" ucd="obs.field" unit="deg" xtype="polygon" value="">
+        <VALUES>
+          <MAX value="81.01558843560966 -69.95461093880266 81.06567094624911 -70.67927922356017 79.34988020864029 -70.684376201833 78.89013903241694 -70.31495449099631 78.91029632344201 -69.95565099163903" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="BAND" datatype="double" arraysize="2" ucd="em.wl;stat.interval" unit="m" xtype="interval" value="">
+        <VALUES>
+          <MIN value="5.800000000000001E-7" />
+          <MAX value="7.8E-7" />
+        </VALUES>
+      </PARAM>
+      <PARAM name="TIME" datatype="double" arraysize="2" ucd="time;stat.interval" unit="d" xtype="interval" value="">
+        <VALUES>
+          <MIN value="50527.527002" />
+          <MAX value="50527.530475" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+</VOTABLE>
diff --git a/pyvo/dal/tests/data/datalink/datalink-obscore.xml b/pyvo/dal/tests/data/datalink/datalink-obscore.xml
new file mode 100644
index 0000000..214805e
--- /dev/null
+++ b/pyvo/dal/tests/data/datalink/datalink-obscore.xml
@@ -0,0 +1,239 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<VOTABLE xmlns="http://www.ivoa.net/xml/VOTable/v1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.4">
+  <RESOURCE type="results">
+    <INFO name="QUERY_STATUS" value="OK" />
+    <INFO name="QUERY_TIMESTAMP" value="2020-03-05T04:21:45.936" />
+    <INFO name="QUERY" value="select * from ivoa.ObsCore" />
+    <TABLE>
+      <FIELD name="calib_level" datatype="int" ucd="meta.code;obs.calib" utype="obscore:ObsDataset.calibLevel">
+        <DESCRIPTION>calibration level (0,1,2,3)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="obs_publisher_did" datatype="char" arraysize="256*" ID="caomPlaneURI" ucd="meta.ref.url;meta.curation" utype="obscore:Curation.PublisherDID">
+        <DESCRIPTION>publisher dataset identifier</DESCRIPTION>
+      </FIELD>
+      <FIELD name="obs_collection" datatype="char" arraysize="128*" ucd="meta.id" utype="obscore:DataID.Collection">
+        <DESCRIPTION>short name for the data colection</DESCRIPTION>
+      </FIELD>
+      <FIELD name="facility_name" datatype="char" arraysize="128*" ucd="meta.id;instr.tel" utype="obscore:Provenance.ObsConfig.Facility.name">
+        <DESCRIPTION>telescope name</DESCRIPTION>
+      </FIELD>
+      <FIELD name="instrument_name" datatype="char" arraysize="128*" ucd="meta.id;instr" utype="obscore:Provenance.ObsConfig.Instrument.name">
+        <DESCRIPTION>instrument name</DESCRIPTION>
+      </FIELD>
+      <FIELD name="obs_id" datatype="char" arraysize="128*" ucd="meta.id" utype="obscore:DataID.observationID">
+        <DESCRIPTION>internal dataset identifier</DESCRIPTION>
+      </FIELD>
+      <FIELD name="dataproduct_type" datatype="char" arraysize="128*" ucd="meta.id" utype="obscore:ObsDataset.dataProductType">
+        <DESCRIPTION>type of product</DESCRIPTION>
+      </FIELD>
+      <FIELD name="obs_release_date" datatype="char" arraysize="23*" ucd="time.release" utype="obscore:Curation.releaseDate" xtype="timestamp">
+        <DESCRIPTION>timestamp of date the data becomes publicly available</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_ra" datatype="double" ucd="pos.eq.ra" unit="deg" utype="obscore:Char.SpatialAxis.Coverage.Location.Coord.Position2D.Value2.C1">
+        <DESCRIPTION>RA of central coordinates</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_dec" datatype="double" ucd="pos.eq.dec" unit="deg" utype="obscore:Char.SpatialAxis.Coverage.Location.Coord.Position2D.Value2.C2">
+        <DESCRIPTION>DEC of central coordinates</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_fov" datatype="double" ucd="phys.angSize;instr.fov" unit="deg" utype="obscore:Char.SpatialAxis.Coverage.Bounds.Extent.diameter">
+        <DESCRIPTION>size of the region covered (~diameter of minimum bounding circle)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_region" datatype="char" arraysize="*" ucd="pos.outline;obs.field" utype="obscore:Char.SpatialAxis.Coverage.Support.Area" xtype="adql:REGION">
+        <DESCRIPTION>region bounded by observation</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_resolution" datatype="double" ucd="pos.angResolution" unit="arcsec" utype="obscore:Char.SpatialAxis.Resolution.refval.value">
+        <DESCRIPTION>typical spatial resolution</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_xel1" datatype="long" ucd="meta.number" utype="obscore:Char.SpatialAxis.numBins1">
+        <DESCRIPTION>dimensions (number of pixels) along one spatial axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="target_name" datatype="char" arraysize="32*" ucd="meta.id;src" utype="obscore:Target.Name">
+        <DESCRIPTION>name of intended target</DESCRIPTION>
+      </FIELD>
+      <FIELD name="s_xel2" datatype="long" ucd="meta.number" utype="obscore:Char.SpatialAxis.numBins2">
+        <DESCRIPTION>dimensions (number of pixels) along the other spatial axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="t_min" datatype="double" ucd="time.start;obs.exposure" unit="d" utype="obscore:Char.TimeAxis.Coverage.Bounds.Limits.StartTime">
+        <DESCRIPTION>start time of observation (MJD)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="t_max" datatype="double" ucd="time.end;obs.exposure" unit="d" utype="obscore:Char.TimeAxis.Coverage.Bounds.Limits.StopTime">
+        <DESCRIPTION>end time of observation (MJD)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="t_exptime" datatype="double" ucd="time.duration;obs.exposure" unit="s" utype="obscore:Char.TimeAxis.Coverage.Support.Extent">
+        <DESCRIPTION>exposure time of observation</DESCRIPTION>
+      </FIELD>
+      <FIELD name="t_resolution" datatype="double" ucd="time.resolution" unit="s" utype="obscore:Char.TimeAxis.Resolution.refval.value">
+        <DESCRIPTION>typical temporal resolution</DESCRIPTION>
+      </FIELD>
+      <FIELD name="t_xel" datatype="long" ucd="meta.number" utype="obscore:Char.TimeAxis.numBins">
+        <DESCRIPTION>dimensions (number of pixels) along the time axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="em_min" datatype="double" ucd="em.wl;stat.min" unit="m" utype="obscore:Char.SpectralAxis.Coverage.Bounds.Limits.LoLimit">
+        <DESCRIPTION>start spectral coordinate value</DESCRIPTION>
+      </FIELD>
+      <FIELD name="em_max" datatype="double" ucd="em.wl;stat.max" unit="m" utype="obscore:Char.SpectralAxis.Coverage.Bounds.Limits.HiLimit">
+        <DESCRIPTION>stop spectral coordinate value</DESCRIPTION>
+      </FIELD>
+      <FIELD name="em_res_power" datatype="double" ucd="spect.resolution" utype="obscore:Char.SpectralAxis.Resolution.ResolPower.refval">
+        <DESCRIPTION>typical spectral resolution</DESCRIPTION>
+      </FIELD>
+      <FIELD name="em_xel" datatype="long" ucd="meta.number" utype="obscore:Char.SpectralAxis.numBins">
+        <DESCRIPTION>dimensions (number of pixels) along the energy axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="pol_xel" datatype="long" ucd="meta.number" utype="obscore:Char.PolarizationAxis.numBins">
+        <DESCRIPTION>dimensions (number of pixels) along the polarization axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="em_ucd" datatype="char" arraysize="32*" ucd="meta.ucd" utype="obscore:Char.SpectralAxis.ucd">
+        <DESCRIPTION>UCD describing the spectral axis</DESCRIPTION>
+      </FIELD>
+      <FIELD name="pol_states" datatype="char" arraysize="32*" ucd="meta.code;phys.polarization" utype="obscore:Char.PolarizationAxis.stateList">
+        <DESCRIPTION>polarization states present in the data</DESCRIPTION>
+      </FIELD>
+      <FIELD name="access_url" datatype="char" arraysize="*" ucd="meta.ref.url" utype="obscore:Access.Reference" xtype="clob">
+        <DESCRIPTION>URL to download the data</DESCRIPTION>
+      </FIELD>
+      <FIELD name="access_estsize" datatype="long" ucd="phys.size;meta.file" unit="kbyte" utype="obscore:Access.Size">
+        <DESCRIPTION>estimated size of the download</DESCRIPTION>
+      </FIELD>
+      <FIELD name="o_ucd" datatype="char" arraysize="32*" ucd="meta.ucd" utype="obscore:Char.ObservableAxis.ucd">
+        <DESCRIPTION>UCD describing the observable axis (pixel values)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="access_format" datatype="char" arraysize="128*" ucd="meta.code.mime" utype="obscore:Access.Format">
+        <DESCRIPTION>format of the data file(s)</DESCRIPTION>
+      </FIELD>
+      <FIELD name="core_id" datatype="char" arraysize="36" xtype="uuid">
+        <DESCRIPTION>primary key</DESCRIPTION>
+      </FIELD>
+      <FIELD name="lastModified" datatype="char" arraysize="23*" xtype="timestamp">
+        <DESCRIPTION>timestamp of last modification of the metadata</DESCRIPTION>
+      </FIELD>
+      <DATA>
+        <TABLEDATA>
+          <TR>
+            <TD>2</TD>
+            <TD>ivo://cadc.nrc.ca/MACHO?54150/cal054150r</TD>
+            <TD>MACHO</TD>
+            <TD>Mt. Stromlo Observatory 50 inch</TD>
+            <TD>MACHO</TD>
+            <TD>54150</TD>
+            <TD>image</TD>
+            <TD>2008-10-08T00:00:00.000</TD>
+            <TD>88.81762854444203</TD>
+            <TD>-67.77109486072844</TD>
+            <TD>1.0245546113974644</TD>
+            <TD>polygon 89.70783791485876 -67.41840881752057 89.74786512306278 -68.14322955012662 88.22311503462467 -68.14813181576002 87.81208257298478 -67.77896886699358 87.82899304985233 -67.4197778334441</TD>
+            <TD>4.353822892</TD>
+            <TD>4181</TD>
+            <TD>macho000071</TD>
+            <TD>4200</TD>
+            <TD>50527.522581</TD>
+            <TD>50527.526053</TD>
+            <TD>300.0</TD>
+            <TD>300.0</TD>
+            <TD>7</TD>
+            <TD>5.800000000000001E-7</TD>
+            <TD>7.8E-7</TD>
+            <TD>5.799999999999999</TD>
+            <TD>7</TD>
+            <TD />
+            <TD />
+            <TD />
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?runid=b7vdfnrcr8kq35a1&amp;ID=ivo%3A%2F%2Fcadc.nrc.ca%2FMACHO%3F54150%2Fcal054150r</TD>
+            <TD />
+            <TD>phot.count</TD>
+            <TD>application/x-votable+xml;content=datalink</TD>
+            <TD>00000000-0000-0000-93cf-fed05e0e60f6</TD>
+            <TD>2019-11-20T19:29:40.929</TD>
+          </TR>
+          <TR>
+            <TD>2</TD>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151b</TD>
+            <TD>MACHO</TD>
+            <TD>Mt. Stromlo Observatory 50 inch</TD>
+            <TD>MACHO</TD>
+            <TD>54151</TD>
+            <TD>image</TD>
+            <TD>2008-10-08T00:00:00.000</TD>
+            <TD>79.9564431173079</TD>
+            <TD>-70.32295142824165</TD>
+            <TD>1.0420934836421956</TD>
+            <TD>polygon 81.03835269676311 -69.95434045997953 81.07118698080274 -70.68523150491481 78.84035765138141 -70.68806135194416 78.88155357298983 -69.9513684574919</TD>
+            <TD>5.020573004</TD>
+            <TD>4175</TD>
+            <TD>macho000006</TD>
+            <TD>4168</TD>
+            <TD>50527.527002</TD>
+            <TD>50527.530475</TD>
+            <TD>300.0</TD>
+            <TD>300.0</TD>
+            <TD>8</TD>
+            <TD>4.800000000000001E-7</TD>
+            <TD>5.800000000000001E-7</TD>
+            <TD>5.799999999999999</TD>
+            <TD>8</TD>
+            <TD />
+            <TD />
+            <TD />
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?runid=b7vdfnrcr8kq35a1&amp;ID=ivo%3A%2F%2Fcadc.nrc.ca%2FMACHO%3F54151%2Fcal054151b</TD>
+            <TD />
+            <TD>phot.count</TD>
+            <TD>application/x-votable+xml;content=datalink</TD>
+            <TD>00000000-0000-0000-b8fb-fed05e2e4e02</TD>
+            <TD>2019-11-20T19:29:40.960</TD>
+          </TR>
+          <TR>
+            <TD>2</TD>
+            <TD>ivo://cadc.nrc.ca/MACHO?54151/cal054151r</TD>
+            <TD>MACHO</TD>
+            <TD>Mt. Stromlo Observatory 50 inch</TD>
+            <TD>MACHO</TD>
+            <TD>54151</TD>
+            <TD>image</TD>
+            <TD>2008-10-08T00:00:00.000</TD>
+            <TD>80.01859124602015</TD>
+            <TD>-70.30750032450052</TD>
+            <TD>1.0248990302163568</TD>
+            <TD>polygon 81.01558843560966 -69.95461093880266 81.06567094624911 -70.67927922356017 79.34988020864029 -70.684376201833 78.89013903241694 -70.31495449099631 78.91029632344201 -69.95565099163903</TD>
+            <TD>5.020573004</TD>
+            <TD>4181</TD>
+            <TD>macho000006</TD>
+            <TD>4169</TD>
+            <TD>50527.527002</TD>
+            <TD>50527.530475</TD>
+            <TD>300.0</TD>
+            <TD>300.0</TD>
+            <TD>7</TD>
+            <TD>5.800000000000001E-7</TD>
+            <TD>7.8E-7</TD>
+            <TD>5.799999999999999</TD>
+            <TD>7</TD>
+            <TD />
+            <TD />
+            <TD />
+            <TD>https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/datalink?runid=b7vdfnrcr8kq35a1&amp;ID=ivo%3A%2F%2Fcadc.nrc.ca%2FMACHO%3F54151%2Fcal054151r</TD>
+            <TD />
+            <TD>phot.count</TD>
+            <TD>application/x-votable+xml;content=datalink</TD>
+            <TD>00000000-0000-0000-b83a-fed05e27f58e</TD>
+            <TD>2019-11-20T19:29:40.960</TD>
+          </TR>
+        </TABLEDATA>
+      </DATA>
+    </TABLE>
+    <INFO name="QUERY_STATUS" value="OVERFLOW" />
+  </RESOURCE>
+  <RESOURCE type="meta" utype="adhoc:service">
+    <PARAM name="resourceIdentifier" datatype="char" arraysize="*" value="ivo://test.com/caom2ops" />
+    <PARAM name="standardID" datatype="char" arraysize="*" value="ivo://ivoa.net/std/DataLink#links-1.0" />
+    <PARAM name="accessURL" datatype="char" arraysize="62" value="https://example.com/obscore-datalink" />
+    <GROUP name="inputParams">
+      <PARAM name="ID" datatype="char" arraysize="*" ref="caomPlaneURI" value="" />
+      <PARAM name="RESPONSEFORMAT" datatype="char" arraysize="*" value="application/x-votable+xml;content=datalink">
+        <VALUES>
+          <OPTION value="application/x-votable+xml;content=datalink" />
+          <OPTION value="application/x-download-manifest+txt" />
+        </VALUES>
+      </PARAM>
+    </GROUP>
+  </RESOURCE>
+</VOTABLE>
diff --git a/pyvo/dal/tests/test_datalink.py b/pyvo/dal/tests/test_datalink.py
index b8f2b63..c771974 100644
--- a/pyvo/dal/tests/test_datalink.py
+++ b/pyvo/dal/tests/test_datalink.py
@@ -32,7 +32,37 @@ def datalink(mocker):
         return get_pkg_data_contents('data/datalink/datalink.xml')
 
     with mocker.register_uri(
-        'GET', 'http://example.com/datalink', content=callback
+        'POST', 'http://example.com/datalink', content=callback
+    ) as matcher:
+        yield matcher
+
+
+@pytest.fixture()
+def obscore_datalink(mocker):
+    def callback(request, context):
+        return get_pkg_data_contents('data/datalink/datalink-obscore.xml')
+
+    with mocker.register_uri(
+        'GET', 'http://example.com/obscore', content=callback
+    ) as matcher:
+        yield matcher
+
+
+@pytest.fixture()
+def res_datalink(mocker):
+
+    first_batch = True
+
+    def callback(request, context):
+        nonlocal first_batch
+        if first_batch:
+            first_batch = False
+            return get_pkg_data_contents('data/datalink/cutout1.xml')
+        else:
+            return get_pkg_data_contents('data/datalink/cutout2.xml')
+
+    with mocker.register_uri(
+        'POST', 'https://example.com/obscore-datalink', content=callback
     ) as matcher:
         yield matcher
 
@@ -59,3 +89,15 @@ def test_datalink():
 
     row = datalink[3]
     assert row.semantics == "#preview"
+
+
+@pytest.mark.usefixtures('obscore_datalink', 'res_datalink')
+@pytest.mark.filterwarnings("ignore::astropy.io.votable.exceptions.W27")
+@pytest.mark.filterwarnings("ignore::astropy.io.votable.exceptions.W06")
+@pytest.mark.filterwarnings("ignore::astropy.io.votable.exceptions.W48")
+@pytest.mark.filterwarnings("ignore::astropy.io.votable.exceptions.E02")
+def test_datalink_batch():
+    results = vo.dal.imagesearch(
+        'http://example.com/obscore', (30, 30))
+
+    assert len([_ for _ in results.iter_datalinks()]) == 3
diff --git a/pyvo/dal/tests/test_params.py b/pyvo/dal/tests/test_params.py
index 63f50f6..800237f 100644
--- a/pyvo/dal/tests/test_params.py
+++ b/pyvo/dal/tests/test_params.py
@@ -7,7 +7,8 @@ from functools import partial
 from urllib.parse import parse_qsl
 
 from pyvo.dal.adhoc import DatalinkResults
-from pyvo.dal.params import find_param_by_keyword, get_converter, AbstractDalQueryParam
+from pyvo.dal.params import find_param_by_keyword, get_converter,\
+    AbstractDalQueryParam
 from pyvo.dal.exceptions import DALServiceError
 
 import pytest
diff --git a/pyvo/dal/tests/test_query.py b/pyvo/dal/tests/test_query.py
index d75b1c9..aa50f1b 100644
--- a/pyvo/dal/tests/test_query.py
+++ b/pyvo/dal/tests/test_query.py
@@ -19,6 +19,7 @@ import numpy as np
 from pyvo.dal.query import DALService, DALQuery, DALResults, Record
 from pyvo.dal.exceptions import DALServiceError, DALQueryError, DALFormatError
 from pyvo.version import version
+from pyvo.utils.compat import ASTROPY_LT_4_1
 
 from astropy.table import Table
 from astropy.io.votable.tree import VOTableFile, Table as VOTable
@@ -123,9 +124,13 @@ def _test_results(results):
     assert results['1', 1] == 42
     assert results['1', 2] == 1337
 
-    assert results['2', 0] == b'Illuminatus'
-    assert results['2', 1] == b"Don't panic, and always carry a towel"
-    assert results['2', 2] == b'Elite'
+    truth = b'Illuminatus' if ASTROPY_LT_4_1 else 'Illuminatus'
+    assert results['2', 0] == truth
+    truth = b"Don't panic, and always carry a towel" \
+        if ASTROPY_LT_4_1 else "Don't panic, and always carry a towel"
+    assert results['2', 1] == truth
+    truth = b'Elite' if ASTROPY_LT_4_1 else 'Elite'
+    assert results['2', 2] == truth
 
 
 def _test_records(records):
@@ -135,13 +140,18 @@ def _test_records(records):
     assert all([isinstance(record, Record) for record in records])
 
     assert records[0]['1'] == 23
-    assert records[0]['2'] == b'Illuminatus'
+
+    truth = b'Illuminatus' if ASTROPY_LT_4_1 else 'Illuminatus'
+    assert records[0]['2'] == truth
 
     assert records[1]['1'] == 42
-    assert records[1]['2'] == b"Don't panic, and always carry a towel"
+    truth = b"Don't panic, and always carry a towel" \
+        if ASTROPY_LT_4_1 else "Don't panic, and always carry a towel"
+    assert records[1]['2'] == truth
 
     assert records[2]['1'] == 1337
-    assert records[2]['2'] == b'Elite'
+    truth = b'Elite' if ASTROPY_LT_4_1 else 'Elite'
+    assert records[2]['2'] == truth
 
 
 @pytest.mark.filterwarnings("ignore::astropy.io.votable.exceptions.W06")
@@ -383,10 +393,11 @@ class TestRecord:
             'http://example.com/query/basic')[0]
 
         assert record['1'] == 23
-        assert record['2'] == b'Illuminatus'
+        truth = b'Illuminatus' if ASTROPY_LT_4_1 else 'Illuminatus'
+        assert record['2'] == truth
 
         assert record['_1'] == 23
-        assert record['_2'] == b'Illuminatus'
+        assert record['_2'] == truth
 
     def test_nosuchcolumn(self):
         record = DALResults.from_result_url(
@@ -413,8 +424,8 @@ class TestRecord:
     def test_repr(self):
         record = DALResults.from_result_url(
             'http://example.com/query/basic')[0]
-
-        assert repr(record) == repr((23, b'Illuminatus'))
+        truth = b'Illuminatus' if ASTROPY_LT_4_1 else 'Illuminatus'
+        assert repr(record) == repr((23, truth))
 
     def test_get(self):
         record = DALResults.from_result_url(
@@ -429,7 +440,8 @@ class TestRecord:
         assert record.getbyucd('foo') == 23
         assert record.getbyucd('bar') == 23
 
-        assert record.getbyutype('foobar') == b'Illuminatus'
+        truth = b'Illuminatus' if ASTROPY_LT_4_1 else 'Illuminatus'
+        assert record.getbyutype('foobar') == truth
 
         record.getbyucd('baz') is None
         record.getbyutype('foobaz') is None
diff --git a/pyvo/dal/tests/test_sia2_remote.py b/pyvo/dal/tests/test_sia2_remote.py
index 54de302..d12475d 100644
--- a/pyvo/dal/tests/test_sia2_remote.py
+++ b/pyvo/dal/tests/test_sia2_remote.py
@@ -25,6 +25,16 @@ class TestSIACadc():
         assert cadc.availability.notes[0] == 'service is accepting queries'
         assert cadc.capabilities
 
+    def test_datalink_batch(self):
+        # Maximum batch size in CADC SIA is around 25
+        # Test whether multiple batches can be retrieved
+        results = search(CADC_SIA_URL, pos=(2.8425, 74.4846, 10), maxrec=55)
+        ids = []
+        for i in results.iter_datalinks():
+            assert i.table[0]['ID'] not in ids
+            ids.append(i.table[0]['ID'])
+        assert len(ids) == 55
+
     def test_pos(self):
         results = search(CADC_SIA_URL, pos=(2.8425, 74.4846, 0.001))
         assert len(results) > 10