Repository: rucio/rucio

Issue:
client_extract=True replicas need to be extracted from the zip file
Motivation
----------
The client needs to check for replicas with client_extract=True and extract them.


Discussion:
I'm going to disable multi threaded download in case that multiple files should be downloaded from the same archive (guess it's not too unlikely for datasets?)

How shall we handle the archive file? I have different approaches in mind:
A
1. download archive
2. extract file
3. go to 1. for next file

B
1. download archive
2. extract file
3. delete archive
4. go to 1. for next file

C
1. download archive if it doesn't exist
2. extract file
3. go to 1. for next file
4. delete all archives

Or any of these approaches with downloading archives into /tmp/... 
How about never downloading the archives into `/tmp`?

1.  Sort all files needed by archive name.  This way, you only download each archive once.
2.
   *  If gfal2 module is present: For each archive needed, read the last 1MB of the archive, parse out the central directory of the zipfile, then read out only the bytes corresponding to files you need.
   *  If gfal2 module is not present: download whole archive to the output directory, then open with python, unlink the file, and start extracting the files needed.

Should be relatively straightforward to avoid downloading the whole archive in the end...
Do I have the archive name?
This ticket should only handle the case when the protocol cannot stream the single file from an archive. I was told to use an external program like unzip in this case. ( @bari12 )
Even if I had the archive name sorting would not be that easy because this would probably break the PFN order
Yes, only the actual local extract case should be handled here, streaming the content of the (uncompressed) zip is already handled in list_replicas.
The result of list_replicas (be it metalink or not) has the PFN of the archive.
Right now the loop is going logical file by file and then do the actual physical download. What you can do (Basically Brians suggestion) is to take the ones which only have replicas saying `client_extract=True` out, remove the duplicates from this list and only download these replicas. Afterwards you extract the files wanted (re-adding the logical files whose duplicate replicas you removed). I know this breaks the current simple "one loop solves it all" concept, but it shouldn't be all too complicated to add.
Can't list_replicas return a file that has some PFNs with client_extract=True and some with False?
So if its always either True or False per file it should make things easier. But in this case client_extract should be a file attribute
Otherwise I would be afraid of cases like:
```
<metalink>
  <file1>
    <url client_extract=True prio=1>same_archive_pfn</url>
    <url client_extract=False prio=2>pfn1</url>
  </file>
  <file2>
    <url client_extract=False prio=1>pfn2</url>
    <url client_extract=True prio=2>same_archive_pfn</url>
  </file>
</metalink>
```
You say in such cases I should skip the prio 1 pfn of file2 ?
@TWAtGH right now, client_extract is set per file (current decision is based purely on if protocol==root)
It should be set per replica. It's a valid case to have a file which has "normal" replicas but is also a constituent of an archive. Thus some replicas of this file would have client_extract=True some would not.

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -29,6 +29,7 @@
 import os
 import os.path
 import random
+import shutil
 import signal
 import time
 
@@ -37,16 +38,80 @@
 except ImportError:
     from queue import Queue, Empty, deque
 from threading import Thread
-from xml.etree import ElementTree
 
 from rucio.client.client import Client
 from rucio.common.exception import (InputValidationError, NoFilesDownloaded, ServiceUnavailable,
                                     NotAllFilesDownloaded, RSENotFound, RucioException, SourceNotFound)
-from rucio.common.utils import adler32, md5, detect_client_location, generate_uuid, send_trace, sizefmt, execute
+from rucio.common.utils import adler32, md5, detect_client_location, generate_uuid, parse_replicas_from_string, send_trace, sizefmt, execute
 from rucio.rse import rsemanager as rsemgr
 from rucio import version
 
 
+class BaseExtractionTool:
+
+    def __init__(self, program_name, useability_check_args, extract_args, logger):
+        """
+        Initialises a extraction tool object
+
+        :param program_name: the name of the archive extraction program, e.g., unzip
+        :param useability_check_args: the arguments of the extraction program to test if its installed, e.g., --version
+        :param extract_args: the arguments that will be passed to the program for extraction
+        :param logger: logging.Logger object
+        """
+        self.program_name = program_name
+        self.useability_check_args = useability_check_args
+        self.extract_args = extract_args
+        self.logger = logger
+        self.is_useable_result = None
+
+    def is_useable(self):
+        """
+        Checks if the extraction tool is installed and usable
+
+        :returns: True if it is usable otherwise False
+        """
+        if self.is_useable_result is not None:
+            return self.is_useable_result
+        self.is_usable_result = False
+        cmd = '%s %s' % (self.program_name, self.useability_check_args)
+        try:
+            exitcode, out, err = execute(cmd)
+            exitcode = int(exitcode)
+            self.logger.debug('"%s" returned with exitcode %d' % (cmd, exitcode))
+            self.is_usable_result = (exitcode == 0)
+        except Exception as error:
+            self.logger.debug('Failed to execute: "%s"' % exitcode)
+            self.logger.debug(error)
+        return self.is_usable_result
+
+    def try_extraction(self, archive_file_path, file_to_extract, dest_dir_path):
+        """
+        Calls the extraction program to extract a file from an archive
+
+        :param archive_file_path: path to the archive
+        :param file_to_extract: file name to extract from the archive
+        :param dest_dir_path: destination directory where the extracted file will be stored
+
+        :returns: True on success otherwise False
+        """
+        if not self.is_useable():
+            return False
+        args_map = {'archive_file_path': archive_file_path,
+                    'file_to_extract': file_to_extract,
+                    'dest_dir_path': dest_dir_path}
+        extract_args = self.extract_args % args_map
+        cmd = '%s %s' % (self.program_name, extract_args)
+        try:
+            exitcode, out, err = execute(cmd)
+            exitcode = int(exitcode)
+            self.logger.debug('"%s" returned with exitcode %d' % (cmd, exitcode))
+            return (exitcode == 0)
+        except Exception as error:
+            self.logger.debug('Failed to execute: "%s"' % exitcode)
+            self.logger.debug(error)
+        return False
+
+
 class DownloadClient:
 
     def __init__(self, client=None, logger=None, tracing=True, check_admin=False):
@@ -69,6 +134,7 @@ def __init__(self, client=None, logger=None, tracing=True, check_admin=False):
 
         self.client_location = detect_client_location()
 
+        self.is_tape_excluded = True
         self.is_admin = False
         if check_admin:
             account_attributes = list(self.client.list_account_attributes(self.client.account))
@@ -77,6 +143,7 @@ def __init__(self, client=None, logger=None, tracing=True, check_admin=False):
                     self.is_admin = attr['value'] is True
                     break
         if self.is_admin:
+            self.is_tape_excluded = False
             logger.debug('Admin mode enabled')
 
         self.trace_tpl = {}
@@ -84,7 +151,18 @@ def __init__(self, client=None, logger=None, tracing=True, check_admin=False):
         self.trace_tpl['localSite'] = self.client_location['site']
         self.trace_tpl['account'] = self.client.account
         self.trace_tpl['eventType'] = 'download'
-        self.trace_tpl['eventVersion'] = 'api_' + version.RUCIO_VERSION[0]
+        self.trace_tpl['eventVersion'] = 'api_%s' % version.RUCIO_VERSION[0]
+
+        self.use_cea_threshold = 10
+        self.extraction_tools = []
+
+        # unzip <archive_file_path> <did_name> -d <dest_dir_path>
+        extract_args = '%(archive_file_path)s %(file_to_extract)s -d %(dest_dir_path)s'
+        self.extraction_tools.append(BaseExtractionTool('unzip', '-v', extract_args, logger))
+
+        # tar -C <dest_dir_path> -xf <archive_file_path>  <did_name>
+        extract_args = '-C %(dest_dir_path)s -xf %(archive_file_path)s %(file_to_extract)s'
+        self.extraction_tools.append(BaseExtractionTool('tar', '--version', extract_args, logger))
 
     def download_file_from_archive(self, items, trace_custom_fields={}):
         """
@@ -250,9 +328,8 @@ def download_pfns(self, items, num_threads=2, trace_custom_fields={}):
             item['name'] = did_name
             item['sources'] = [{'pfn': pfn, 'rse': rse}]
             dest_file_path = os.path.join(dest_dir_path, did_name)
-            item['dest_dir_path'] = dest_dir_path
-            item['dest_file_path'] = dest_file_path
-            item['temp_file_path'] = dest_file_path + '.part'
+            item['dest_file_paths'] = [dest_file_path]
+            item['temp_file_path'] = '%s.part' % dest_file_path
             item.setdefault('ignore_checksum', True)
 
             input_items.append(item)
@@ -266,14 +343,16 @@ def download_pfns(self, items, num_threads=2, trace_custom_fields={}):
 
         return self._check_output(output_items)
 
-    def download_dids(self, items, num_threads=2, trace_custom_fields={}, filters={}):
+    def download_dids(self, items, num_threads=2, trace_custom_fields={}):
         """
         Download items with given DIDs. This function can also download datasets and wildcarded DIDs.
 
         :param items: List of dictionaries. Each dictionary describing an item to download. Keys:
-            did                 - DID string of this file (e.g. 'scope:file.name'). Wildcards are not allowed
+            did                 - DID string of this file (e.g. 'scope:file.name')
+            filters             - Filter to select DIDs for download. Optional if DID is given
             rse                 - Optional: rse name (e.g. 'CERN-PROD_DATADISK') or rse expression from where to download
-            resolve_archives    - Optional: bool indicating whether archives should be considered for download (Default: False)
+            no_resolve_archives - Optional: bool indicating whether archives should not be considered for download (Default: False)
+            resolve_archives    - Deprecated: Use no_resolve_archives instead
             force_scheme        - Optional: force a specific scheme to download this item. (Default: None)
             base_dir            - Optional: base directory where the downloaded files will be stored. (Default: '.')
             no_subdir           - Optional: If true, files are written directly into base_dir and existing files are overwritten. (Default: False)
@@ -282,7 +361,6 @@ def download_dids(self, items, num_threads=2, trace_custom_fields={}, filters={}
             transfer_timeout    - Optional: Timeout time for the download protocols. (Default: None)
         :param num_threads: Suggestion of number of threads to use for the download. It will be lowered if it's too high.
         :param trace_custom_fields: Custom key value pairs to send with the traces.
-        :param filters: dictionary containing filter options.
 
         :returns: a list of dictionaries with an entry for each file, containing the input options, the did, and the clientState
 
@@ -291,9 +369,18 @@ def download_dids(self, items, num_threads=2, trace_custom_fields={}, filters={}
         :raises NotAllFilesDownloaded: if not all files could be downloaded
         :raises RucioException: if something unexpected went wrong during the download
         """
+        logger = self.logger
         trace_custom_fields['uuid'] = generate_uuid()
 
-        input_items = self._prepare_items_for_download(items, filters)
+        logger.info('Processing %d item(s) for input' % len(items))
+        download_info = self._resolve_and_merge_input_items(copy.deepcopy(items))
+        did_to_options = download_info['did_to_options']
+        merged_items = download_info['merged_items']
+
+        self.logger.debug('num_unmerged_items=%d; num_dids=%d; num_merged_items=%d' % (len(items), len(did_to_options), len(merged_items)))
+
+        logger.info('Getting sources of DIDs')
+        input_items = self._prepare_items_for_download(did_to_options, merged_items)
 
         num_files_in = len(input_items)
         output_items = self._download_multithreaded(input_items, num_threads, trace_custom_fields)
@@ -401,7 +488,6 @@ def _download_item(self, item, trace, log_prefix=''):
         :returns: dictionary with all attributes from the input item and a clientState attribute
         """
         logger = self.logger
-
         did_scope = item['scope']
         did_name = item['name']
         did_str = '%s:%s' % (did_scope, did_name)
@@ -414,18 +500,22 @@ def _download_item(self, item, trace, log_prefix=''):
         trace.setdefault('dataset', item.get('dataset_name', ''))
         trace.setdefault('filesize', item.get('bytes'))
 
-        # if file already exists, set state, send trace, and return
-        temp_file_path = item['temp_file_path']
-        dest_file_path = item['dest_file_path']
-        if os.path.isfile(dest_file_path):
-            logger.info('%sFile exists already locally: %s' % (log_prefix, did_str))
-            item['clientState'] = 'ALREADY_DONE'
+        dest_file_paths = item['dest_file_paths']
 
-            trace['transferStart'] = time.time()
-            trace['transferEnd'] = time.time()
-            trace['clientState'] = 'ALREADY_DONE'
-            self._send_trace(trace)
-            return item
+        # if file already exists make sure it exists at all destination paths, set state, send trace, and return
+        for dest_file_path in dest_file_paths:
+            if os.path.isfile(dest_file_path):
+                logger.info('%sFile exists already locally: %s' % (log_prefix, did_str))
+                for missing_file_path in dest_file_paths:
+                    if not os.path.isfile(missing_file_path):
+                        logger.debug("copying '%s' to '%s'" % (dest_file_path, missing_file_path))
+                        shutil.copy2(dest_file_path, missing_file_path)
+                item['clientState'] = 'ALREADY_DONE'
+                trace['transferStart'] = time.time()
+                trace['transferEnd'] = time.time()
+                trace['clientState'] = 'ALREADY_DONE'
+                send_trace(trace, self.client.host, self.client.user_agent)
+                return item
 
         # check if file has replicas
         sources = item.get('sources')
@@ -441,15 +531,16 @@ def _download_item(self, item, trace, log_prefix=''):
         # try different PFNs until one succeeded
         i = 0
         while not success and i < len(sources):
-            pfn = sources[i]['pfn']
-            rse_name = sources[i]['rse']
+            source = sources[i]
             i += 1
+            pfn = source['pfn']
+            rse_name = source['rse']
             scheme = pfn.split(':')[0]
 
             # this is a workaround to fix that gfal doesnt use root's -z option for archives
             # this will be removed as soon as gfal has fixed this
             temp_file_path = item['temp_file_path']
-            dest_file_path = item['dest_file_path']
+            dest_file_path = item['dest_file_paths'][0]
             unzip_arg_name = '?xrdcl.unzip='
             if scheme == 'root' and unzip_arg_name in pfn:
                 logger.info('%sFound xrdcl.unzip in PFN. Using xrdcp overwrite.' % log_prefix)
@@ -555,8 +646,13 @@ def _download_item(self, item, trace, log_prefix=''):
             item['clientState'] = 'FAILED'
             return item
 
-        logger.debug("renaming '%s' to '%s'" % (temp_file_path, dest_file_path))
-        os.rename(temp_file_path, dest_file_path)
+        dest_file_path_iter = iter(dest_file_paths)
+        first_dest_file_path = next(dest_file_path_iter)
+        logger.debug("renaming '%s' to '%s'" % (temp_file_path, first_dest_file_path))
+        os.rename(temp_file_path, first_dest_file_path)
+        for cur_dest_file_path in dest_file_path_iter:
+            logger.debug("copying '%s' to '%s'" % (first_dest_file_path, cur_dest_file_path))
+            shutil.copy2(first_dest_file_path, cur_dest_file_path)
 
         trace['transferStart'] = start_time
         trace['transferEnd'] = end_time
@@ -572,6 +668,37 @@ def _download_item(self, item, trace, log_prefix=''):
             logger.info('%sFile %s successfully downloaded. %s in %s seconds = %s MBps' % (log_prefix, did_str, size_str, duration, rate))
         else:
             logger.info('%sFile %s successfully downloaded in %s seconds' % (log_prefix, did_str, duration))
+
+        file_items_in_archive = item.get('archive_items', [])
+        if len(file_items_in_archive) > 0:
+            logger.info('%sExtracting %d file(s) from %s' % (log_prefix, len(file_items_in_archive), did_name))
+
+            archive_file_path = first_dest_file_path
+            for file_item in file_items_in_archive:
+                extraction_ok = False
+                extract_file_name = file_item['name']
+                dest_file_path_iter = iter(file_item['dest_file_paths'])
+                first_dest_file_path = next(dest_file_path_iter)
+                dest_dir = os.path.dirname(first_dest_file_path)
+                logger.debug('%sExtracting %s to %s' % (log_prefix, extract_file_name, dest_dir))
+                for extraction_tool in self.extraction_tools:
+                    if extraction_tool.try_extraction(archive_file_path, extract_file_name, dest_dir):
+                        extraction_ok = True
+                        break
+
+                if not extraction_ok:
+                    logger.error('Extraction of file %s from archive %s failed.' % (extract_file_name, did_name))
+                    continue
+
+                first_dest_file_path = os.path.join(dest_dir, extract_file_name)
+                for cur_dest_file_path in dest_file_path_iter:
+                    logger.debug("copying '%s' to '%s'" % (first_dest_file_path, cur_dest_file_path))
+                    shutil.copy2(first_dest_file_path, cur_dest_file_path)
+
+            if not item.get('shall_keep_archive'):
+                logger.debug('%sDeleting archive %s' % (log_prefix, did_name))
+                os.remove(archive_file_path)
+
         return item
 
     def download_aria2c(self, items, trace_custom_fields={}, filters={}):
@@ -597,15 +724,26 @@ def download_aria2c(self, items, trace_custom_fields={}, filters={}):
         :raises NotAllFilesDownloaded: if not all files could be downloaded
         :raises RucioException: if something went wrong during the download (e.g. aria2c could not be started)
         """
+        logger = self.logger
         trace_custom_fields['uuid'] = generate_uuid()
 
         rpc_secret = '%x' % (random.getrandbits(64))
-        rpc_auth = 'token:' + rpc_secret
+        rpc_auth = 'token:%s' % rpc_secret
         rpcproc, aria_rpc = self._start_aria2c_rpc(rpc_secret)
 
         for item in items:
             item['force_scheme'] = ['https', 'davs']
-        input_items = self._prepare_items_for_download(items, filters)
+            item['no_resolve_archives'] = True
+
+        logger.info('Processing %d item(s) for input' % len(items))
+        download_info = self._resolve_and_merge_input_items(copy.deepcopy(items))
+        did_to_options = download_info['did_to_options']
+        merged_items = download_info['merged_items']
+
+        self.logger.debug('num_unmerged_items=%d; num_dids=%d; num_merged_items=%d' % (len(items), len(did_to_options), len(merged_items)))
+
+        logger.info('Getting sources of DIDs')
+        input_items = self._prepare_items_for_download(did_to_options, merged_items)
 
         try:
             output_items = self._download_items_aria2c(input_items, aria_rpc, rpc_auth, trace_custom_fields)
@@ -739,7 +877,9 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
                     pfn_to_rse[pfn] = src['rse']
 
                 # does file exist and are sources available?
-                if os.path.isfile(item['dest_file_path']):
+                # workaround: only consider first dest file path for aria2c download
+                dest_file_path = next(iter(item['dest_file_paths']))
+                if os.path.isfile(dest_file_path):
                     logger.info('File exists already locally: %s' % file_did_str)
                     item['clientState'] = 'ALREADY_DONE'
                     trace['clientState'] = 'ALREADY_DONE'
@@ -751,7 +891,7 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
                     self._send_trace(trace)
                 else:
                     item['trace'] = trace
-                    options = {'dir': item['dest_dir_path'],
+                    options = {'dir': os.path.dirname(dest_file_path),
                                'out': os.path.basename(item['temp_file_path'])}
                     gid = aria_rpc.aria2.addUri(rpc_auth, pfns, options)
                     gid_to_item[gid] = item
@@ -780,7 +920,8 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
                 file_name = item['name']
                 file_did_str = '%s:%s' % (file_scope, file_name)
                 temp_file_path = item['temp_file_path']
-                dest_file_path = item['dest_file_path']
+                # workaround: only consider first dest file path for aria2c download
+                dest_file_path = next(iter(item['dest_file_paths']))
 
                 # ensure we didnt miss the active state (e.g. a very fast download)
                 start_time = item.setdefault('transferStart', time.time())
@@ -841,133 +982,388 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
 
         return items
 
-    def _prepare_items_for_download(self, items, filters={}):
+    def _resolve_and_merge_input_items(self, items):
         """
-        Resolves wildcarded DIDs, get DID details (e.g. type), and collects
-        the available replicas for each DID
-        (This function is meant to be used as class internal only)
+        This function takes the input items given to download_dids etc. and merges them
+        respecting their individual options. This way functions can operate on these items
+        in batch mode. E.g., list_replicas calls are reduced.
 
-        :param items: list of dictionaries containing the items to prepare
-        :param filters: dictionary containing filters
+        :param items: List of dictionaries. Each dictionary describing an input item
 
-        :returns: list of dictionaries, one dict for each file to download
+        :returns: a dictionary with a dictionary that maps the input DIDs to options
+                  and a list with a dictionary for each merged download item
 
-        :raises InputValidationError: if the given input is not valid or incomplete
+        :raises InputValidationError: if one of the input items is in the wrong format
         """
         logger = self.logger
 
-        logger.info('Processing %d item(s) for input' % len(items))
-        resolved_items = []
-        # resolve input: extend rse expression, resolve wildcards, get did type
+        # check mandatory options before doing any server calls
         for item in items:
-            did_str = item.get('did')
-            if not did_str and not filters:
-                raise InputValidationError('The key did is mandatory')
-
-            logger.debug('Processing item %s' % did_str)
-
-            new_item = copy.deepcopy(item)
-
-            # extend RSE expression to exclude tape RSEs for non-admin accounts
-            if not self.is_admin:
-                rse = new_item.get('rse')
-                new_item['rse'] = 'istape=False' if not rse else '(%s)&istape=False' % rse
-                logger.debug('RSE-Expression: %s' % new_item['rse'])
-
-            if not did_str:
-                logger.debug('Resolving DIDs by using filter options.')
-                scope = filters['scope']
-                del filters['scope']
-                for dids in self.client.list_dids(scope, filters=filters, type='all', long=True):
-                    logger.debug('%s - %s:%s' % (dids['did_type'], scope, dids['name']))
-                    new_item['scope'] = scope
-                    new_item['type'] = dids['did_type'].upper()
-                    new_item['name'] = dids['name']
-                    new_item['did'] = '%s:%s' % (scope, dids['name'])
-                    resolved_items.append(copy.deepcopy(new_item))
-            else:
-                # resolve any wildcards in the input dids
+            if item.get('resolve_archives') is not None:
+                logger.warning('resolve_archives option is deprecated and will be removed in a future release.')
+                item.setdefault('no_resolve_archives', not item.pop('resolve_archives'))
+
+            did = item.get('did', [])
+            if len(did) == 0:
+                if not item.get('filters', {}).get('scope'):
+                    logger.debug(item)
+                    raise InputValidationError('Item without did and filter/scope')
+                item['did'] = [None]
+            elif not isinstance(did, list):
+                item['did'] = [did]
+
+        distinct_keys = ['rse', 'force_scheme', 'nrandom']
+        all_resolved_did_strs = set()
+
+        did_to_options = {}
+        merged_items = []
+        download_info = {'did_to_options': did_to_options,
+                         'merged_items': merged_items}
+
+        while len(items) > 0:
+            item = items.pop()
+
+            filters = item.get('filters')
+            item_dids = item.pop('did')
+            if item_dids[0] is None:
+                logger.debug('Resolving DIDs by using filter options')
+                item_dids = []
+                scope = filters.pop('scope')
+                for did_name in self.client.list_dids(scope, filters=filters, type='all'):
+                    item_dids.append('%s:%s' % (scope, did_name))
+
+            base_dir = item.pop('base_dir', '.')
+            no_subdir = item.pop('no_subdir', False)
+            ignore_checksum = item.pop('ignore_checksum', False)
+            new_transfer_timeout = item.pop('transfer_timeout', None)
+            resolved_dids = item.setdefault('dids', [])
+            for did_str in item_dids:
                 did_scope, did_name = self._split_did_str(did_str)
-                logger.debug('Splitted DID: %s:%s' % (did_scope, did_name))
-                new_item['scope'] = did_scope
+                tmp_did_names = []
                 if '*' in did_name:
-                    logger.debug('Resolving wildcarded DID %s' % did_str)
                     filters['name'] = did_name
-                    for dids in self.client.list_dids(did_scope, filters=filters, type='all', long=True):
-                        logger.debug('%s - %s:%s' % (dids['did_type'], did_scope, dids['name']))
-                        new_item['type'] = dids['did_type'].upper()
-                        new_item['name'] = dids['name']
-                        new_item['did'] = '%s:%s' % (did_scope, dids['name'])
-                        resolved_items.append(copy.deepcopy(new_item))
+                    tmp_did_names = list(self.client.list_dids(did_scope, filters=filters, type='all'))
                 else:
-                    new_item['type'] = self.client.get_did(did_scope, did_name)['type'].upper()
-                    new_item['name'] = did_name
-                    resolved_items.append(new_item)
+                    tmp_did_names = [did_name]
+
+                for did_name in tmp_did_names:
+                    resolved_did_str = '%s:%s' % (did_scope, did_name)
+                    options = did_to_options.setdefault(resolved_did_str, {})
+                    options.setdefault('destinations', set()).add((base_dir, no_subdir))
+
+                    if resolved_did_str in all_resolved_did_strs:
+                        # in this case the DID was already given in another item
+                        # the options of this DID will be ignored and the options of the first item that contained the DID will be used
+                        # another approach would be to compare the options and apply the more relaxed options
+                        logger.debug('Ignoring further options of DID: %s' % resolved_did_str)
+                        continue
+
+                    options['ignore_checksum'] = (options.get('ignore_checksum') or ignore_checksum)
+                    cur_transfer_timeout = options.setdefault('transfer_timeout', None)
+                    if cur_transfer_timeout is not None and new_transfer_timeout is not None:
+                        options['transfer_timeout'] = max(int(cur_transfer_timeout), int(new_transfer_timeout))
+                    elif new_transfer_timeout is not None:
+                        options['transfer_timeout'] = int(new_transfer_timeout)
+
+                    resolved_dids.append({'scope': did_scope, 'name': did_name})
+                    all_resolved_did_strs.add(resolved_did_str)
+
+            if len(resolved_dids) == 0:
+                logger.warning('An item didnt have any DIDs after resolving the input. Ignoring it.')
+                logger.debug(item)
+                continue
 
-        # this list will have one dict for each file to download
-        file_items = []
-        # get replicas for every file of the given dids
-        logger.debug('%d DIDs after processing input' % len(resolved_items))
-        for item in resolved_items:
-            did_scope = item['scope']
-            did_name = item['name']
-            did_str = item['did']
+            was_merged = False
+            for merged_item in merged_items:
+                if all(item.get(k) == merged_item.get(k) for k in distinct_keys):
+                    merged_item['dids'].extend(resolved_dids)
+                    was_merged = True
+                    break
+            if not was_merged:
+                item['dids'] = resolved_dids
+                merged_items.append(item)
+        return download_info
+
+    def _prepare_items_for_download(self, did_to_options, merged_items):
+        """
+        Get sources (PFNs) of the DIDs and optimises the amount of files to download
+        (This function is meant to be used as class internal only)
+
+        :param did_to_options: dictionary that maps each input DID to some input options
+        :param merged_items: list of dictionaries. Each dictionary describes a bunch of DIDs to download
+
+        :returns: list of dictionaries. Each dictionary describes an element to download
+
+        :raises InputValidationError: if the given input is not valid or incomplete
+        """
+        logger = self.logger
 
-            logger.debug('Processing: %s' % item)
+        # if one item wants to resolve archives we enable it for all items
+        resolve_archives = not all(item.get('no_resolve_archives') for item in merged_items)
+        if resolve_archives:
+            # perhaps we'll need an extraction tool so check what is installed
+            self.extraction_tools = [tool for tool in self.extraction_tools if tool.is_useable()]
+            if len(self.extraction_tools) < 1:
+                logger.warning('Archive resolution is enabled but no extraction tool is available. '
+                               'Sources whose protocol doesnt support extraction wont be considered for download.')
 
-            # since we are using metalink we need to explicitly
-            # give all schemes (probably due to a bad server site implementation)
+        # maps file item IDs (fiid) to the file item object
+        fiid_to_file_item = {}
+
+        # list of all file item objects
+        all_file_items = []
+
+        # cea -> client_extract archives to avoid confusion with archives that dont need explicit extraction
+        # this dict will contain all ids of cea's that definitely will be downloaded
+        cea_id_pure_to_fiids = {}
+
+        # this dict will contain ids of cea's that have higher prioritised non cea sources
+        cea_id_mixed_to_fiids = {}
+
+        all_input_dids = set(did_to_options.keys())
+        all_dest_file_paths = set()
+
+        # get replicas for every file of the given dids
+        logger.debug('num list_replicas calls: %d' % len(merged_items))
+        for item in merged_items:
+            # since we're using metalink we need to explicitly give all schemes
             force_scheme = item.get('force_scheme')
             if force_scheme:
                 schemes = force_scheme if isinstance(force_scheme, list) else [force_scheme]
             else:
                 schemes = ['davs', 'gsiftp', 'https', 'root', 'srm', 'file']
+            logger.debug('schemes: %s' % schemes)
+
+            # extend RSE expression to exclude tape RSEs for non-admin accounts
+            rse_expression = item.get('rse')
+            if self.is_tape_excluded:
+                rse_expression = 'istape=False' if not rse_expression else '(%s)&istape=False' % rse_expression
+            logger.debug('rse_expression: %s' % rse_expression)
 
             # get PFNs of files and datasets
-            metalink_str = self.client.list_replicas([{'scope': did_scope, 'name': did_name}],
+            logger.debug('num DIDs for list_replicas call: %d' % len(item['dids']))
+            metalink_str = self.client.list_replicas(item['dids'],
                                                      schemes=schemes,
-                                                     rse_expression=item.get('rse'),
+                                                     rse_expression=rse_expression,
                                                      client_location=self.client_location,
-                                                     resolve_archives=item.get('resolve_archives', False),
+                                                     resolve_archives=resolve_archives,
+                                                     resolve_parents=True,
                                                      metalink=True)
-            files_with_pfns = self._parse_list_replica_metalink(metalink_str)
+            file_items = parse_replicas_from_string(metalink_str)
+
+            logger.debug('num resolved files: %s' % len(file_items))
 
             nrandom = item.get('nrandom')
             if nrandom:
-                logger.info('Selecting %d random replicas from dataset %s' % (nrandom, did_str))
-                random.shuffle(files_with_pfns)
-                files_with_pfns = files_with_pfns[0:nrandom]
+                logger.info('Selecting %d random replicas from DID(s): %s' % (nrandom, item['dids']))
+                random.shuffle(file_items)
+                file_items = file_items[0:nrandom]
 
-            for file_item in files_with_pfns:
-                file_did_scope = file_item['scope']
-                file_did_name = file_item['name']
-                file_did_str = '%s:%s' % (file_did_scope, file_did_name)
+            all_file_items.extend(file_items)
+
+            for file_item in file_items:
+                # parent_dids contains all parents, so we take the intersection with the input dids
+                dataset_did_strs = file_item.setdefault('parent_dids', set())
+                dataset_did_strs.intersection_update(all_input_dids)
+
+                file_did_str = file_item['did']
+                file_did_scope, file_did_name = self._split_did_str(file_did_str)
+                file_item['scope'] = file_did_scope
+                file_item['name'] = file_did_name
 
                 logger.debug('Queueing file: %s' % file_did_str)
+                logger.debug('real parents: %s' % dataset_did_strs)
+                logger.debug('options: %s' % did_to_options)
+
+                # prepare destinations:
+                # if datasets were given: prepare the destination paths for each dataset
+                options = None
+                dest_file_paths = file_item.get('dest_file_paths', set())
+                for dataset_did_str in dataset_did_strs:
+                    options = did_to_options.get(dataset_did_str)
+                    if not options:
+                        logger.error('No input options available for %s' % dataset_did_str)
+                        continue
+
+                    destinations = options['destinations']
+                    dataset_scope, dataset_name = self._split_did_str(dataset_did_str)
+                    paths = [os.path.join(self._prepare_dest_dir(dest[0], dataset_name, file_did_name, dest[1]), file_did_name) for dest in destinations]
+                    if any(path in all_dest_file_paths for path in paths):
+                        raise RucioException("Multiple file items with same destination file path")
+
+                    all_dest_file_paths.update(paths)
+                    dest_file_paths.update(paths)
+
+                    # workaround: just take any given dataset for the traces and the output
+                    file_item.setdefault('dataset_scope', dataset_scope)
+                    file_item.setdefault('dataset_name', dataset_name)
+
+                # if no datasets were given only prepare the given destination paths
+                if len(dataset_did_strs) == 0:
+                    options = did_to_options.get(file_did_str)
+                    if not options:
+                        logger.error('No input options available for %s' % file_did_str)
+                        continue
+                    destinations = options['destinations']
+                    paths = [os.path.join(self._prepare_dest_dir(dest[0], file_did_scope, file_did_name, dest[1]), file_did_name) for dest in destinations]
+                    if any(path in all_dest_file_paths for path in paths):
+                        raise RucioException("Multiple file items with same destination file path")
+                    all_dest_file_paths.update(paths)
+                    dest_file_paths.update(paths)
+
+                if options is None:
+                    continue
+                file_item['merged_options'] = options
+                file_item['dest_file_paths'] = list(dest_file_paths)
+                file_item['temp_file_path'] = '%s.part' % file_item['dest_file_paths'][0]
+
+                # the file did str ist not an unique key for this dict because multiple calls of list_replicas
+                # could result in the same DID multiple times. So we're using the id of the dictionary objects
+                fiid = id(file_item)
+                fiid_to_file_item[fiid] = file_item
+
+                if resolve_archives:
+                    min_cea_priority = None
+                    num_non_cea_sources = 0
+                    cea_ids = []
+                    sources = []
+                    # go through sources and check how many (non-)cea sources there are,
+                    # index cea sources, or remove cea sources if there is no extraction tool
+                    for source in file_item['sources']:
+                        is_cea = source.get('client_extract', False)
+                        if is_cea and (len(self.extraction_tools) > 0):
+                            priority = int(source['priority'])
+                            if min_cea_priority is None or priority < min_cea_priority:
+                                min_cea_priority = priority
+
+                            # workaround since we dont have the archive DID use the part behind the last slash of the PFN
+                            # this doesn't respect the scope of the archive DID!!!
+                            # and we trust that client_extract==True sources dont have any parameters at the end of the PFN
+                            cea_id = source['pfn'].split('/')
+                            cea_id = cea_id[-1] if len(cea_id[-1]) > 0 else cea_id[-2]
+                            cea_ids.append(cea_id)
+
+                            sources.append(source)
+                        elif not is_cea:
+                            num_non_cea_sources += 1
+                            sources.append(source)
+                        else:
+                            # no extraction tool
+                            logger.debug('client_extract=True; ignoring source: %s' % source['pfn'])
+
+                    logger.debug('Prepared sources: num_sources=%d/%d; num_non_cea_sources=%d; num_cea_ids=%d'
+                                 % (len(sources), len(file_item['sources']), num_non_cea_sources, len(cea_ids)))
+
+                    file_item['sources'] = sources
+
+                    # decide if file item belongs to the pure or mixed map
+                    # if no non-archive src exists or the highest prio src is an archive src we put it in the pure map
+                    if num_non_cea_sources == 0 or min_cea_priority == 1:
+                        logger.debug('Adding fiid to cea pure map: '
+                                     'num_non_cea_sources=%d; min_cea_priority=%d; num_cea_sources=%d'
+                                     % (num_non_cea_sources, min_cea_priority, len(cea_ids)))
+                        for cea_id in cea_ids:
+                            cea_id_pure_to_fiids.setdefault(cea_id, set()).add(fiid)
+                            file_item.setdefault('cea_ids_pure', set()).add(cea_id)
+                    # if there are non-archive sources and archive sources we put it in the mixed map
+                    elif len(cea_ids) > 0:
+                        logger.debug('Adding fiid to cea mixed map: '
+                                     'num_non_cea_sources=%d; min_cea_priority=%d; num_cea_sources=%d'
+                                     % (num_non_cea_sources, min_cea_priority, len(cea_ids)))
+                        for cea_id in cea_ids:
+                            cea_id_mixed_to_fiids.setdefault(cea_id, set()).add(fiid)
+                            file_item.setdefault('cea_ids_mixed', set()).add(cea_id)
+
+        # put all archives from the mixed list into the pure list if they meet
+        # certain conditions, e.g., an archive that is already in the pure list
+        for cea_id_mixed in list(cea_id_mixed_to_fiids.keys()):
+            fiids_mixed = cea_id_mixed_to_fiids[cea_id_mixed]
+            if cea_id_mixed in cea_id_pure_to_fiids:
+                # file from mixed list is already in a pure list
+                logger.debug('Mixed ID is already in cea pure map: '
+                             'cea_id_mixed=%s; num_fiids_mixed=%d; num_cea_pure_fiids=%d'
+                             % (cea_id_mixed, len(fiids_mixed), len(cea_id_pure_to_fiids[cea_id_mixed])))
+            elif len(fiids_mixed) >= self.use_cea_threshold:
+                # more than use_cea_threshold files are in a common archive
+                logger.debug('Number of needed files in cea reached threshold: '
+                             'cea_id_mixed=%s; num_fiids_mixed=%d; threshold=%d'
+                             % (cea_id_mixed, len(fiids_mixed), self.use_cea_threshold))
+            else:
+                # dont move from mixed list to pure list
+                continue
 
-                # put the input options from item into the file item
-                file_item.update(item)
-
-                dest_dir_name = file_did_scope
-                if item['type'] != 'FILE':
-                    # if the did is a dataset, scope and name were updated wrongly
-                    file_item['scope'] = file_did_scope
-                    file_item['name'] = file_did_name
-                    file_item['did'] = file_did_str
-                    file_item['dataset_scope'] = did_scope
-                    file_item['dataset_name'] = did_name
-                    dest_dir_name = did_name
-
-                dest_dir_path = self._prepare_dest_dir(item.get('base_dir', '.'),
-                                                       dest_dir_name, file_did_name,
-                                                       item.get('no_subdir'))
-                file_item['dest_dir_path'] = dest_dir_path
-                dest_file_path = os.path.join(dest_dir_path, file_did_name)
-                file_item['dest_file_path'] = dest_file_path
-                file_item['temp_file_path'] = dest_file_path + '.part'
-
-                file_items.append(file_item)
-        return file_items
+            # remove mixed cea id from mixed map and add it to pure map
+            cea_id_mixed_to_fiids.pop(cea_id_mixed)
+            cea_id_pure_to_fiids.setdefault(cea_id_mixed, set()).update(fiids_mixed)
+
+            # move file items from mixed list to pure list
+            for fiid_mixed in fiids_mixed:
+                file_item = fiid_to_file_item[fiid_mixed]
+                file_item.setdefault('cea_ids_pure', set()).add(cea_id_mixed)
+
+                # remove references from file to mixed archive and from mixed archive to file
+                for cea_id_mixed in file_item.pop('cea_ids_mixed'):
+                    cea_id_mixed_to_fiids[cea_id_mixed].discard(fiid_mixed)
+                    if not len(cea_id_mixed_to_fiids[cea_id_mixed]):
+                        cea_id_mixed_to_fiids.pop(cea_id_mixed)
+
+        for file_item in all_file_items:
+            cea_ids_pure = file_item.get('cea_ids_pure', set())
+            cea_ids_mixed = file_item.get('cea_ids_mixed', set())
+
+            if len(cea_ids_pure) > 0:
+                logger.debug('Removing all non-cea sources of file %s' % file_item['did'])
+                file_item['sources'] = [s for s in file_item['sources'] if s.get('client_extract', False)]
+            elif len(cea_ids_mixed) > 0:
+                logger.debug('Removing all cea sources of file %s' % file_item['did'])
+                file_item['sources'] = [s for s in file_item['sources'] if not s.get('client_extract', False)]
+
+        # reduce the amount of archives to download by removing
+        # all redundant pure archives (=all files can be extracted from other archives)
+        for cea_id_pure in list(cea_id_pure_to_fiids.keys()):
+            # if all files of this archive are available in more than one archive the archive is redundant
+            if all(len(fiid_to_file_item[fiid_pure]['cea_ids_pure']) > 1 for fiid_pure in cea_id_pure_to_fiids[cea_id_pure]):
+                for fiid_pure in cea_id_pure_to_fiids[cea_id_pure]:
+                    fiid_to_file_item[fiid_pure]['cea_ids_pure'].discard(cea_id_pure)
+                logger.debug('Removing redundant archive %s' % cea_id_pure)
+                cea_id_pure_to_fiids.pop(cea_id_pure)
+
+        # remove all archives of a file except a single one so
+        # that each file is assigned to exactly one pure archive
+        for cea_id_pure in cea_id_pure_to_fiids:
+            for fiid_pure in cea_id_pure_to_fiids[cea_id_pure]:
+                cea_ids_pure = fiid_to_file_item[fiid_pure]['cea_ids_pure']
+                for cea_id_pure_other in list(cea_ids_pure):
+                    if cea_id_pure != cea_id_pure_other:
+                        cea_id_pure_to_fiids[cea_id_pure_other].discard(fiid_pure)
+                        cea_ids_pure.discard(cea_id_pure_other)
+
+        download_packs = []
+        cea_id_to_pack = {}
+        for file_item in all_file_items:
+            cea_ids = file_item.get('cea_ids_pure', set())
+            if len(cea_ids) > 0:
+                cea_id = next(iter(cea_ids))
+                pack = cea_id_to_pack.get(cea_id)
+                if pack is None:
+                    scope = file_item['scope']
+                    first_dest = next(iter(file_item['merged_options']['destinations']))
+                    dest_path = os.path.join(self._prepare_dest_dir(first_dest[0], scope, cea_id, first_dest[1]), cea_id)
+                    pack = {'scope': scope,
+                            'name': cea_id,
+                            'dest_file_paths': [dest_path],
+                            'temp_file_path': '%s.part' % dest_path,
+                            'sources': file_item['sources'],
+                            'ignore_checksum': True,  # we currently dont have checksums for the archive
+                            'archive_items': []
+                            }
+                    cea_id_to_pack[cea_id] = pack
+                    download_packs.append(pack)
+                file_item.pop('sources')
+                pack['archive_items'].append(file_item)
+            else:
+                download_packs.append(file_item)
+        return download_packs
 
     def _split_did_str(self, did_str):
         """
@@ -998,61 +1394,6 @@ def _split_did_str(self, did_str):
 
         return did_scope, did_name
 
-    def _parse_list_replica_metalink(self, metalink_str):
-        """
-        Parses the metalink string that list_replicas can return into a list of dictionaries.
-        (This function is meant to be used as class internal only)
-
-        :param metalink_str: the metalink string to be parsed
-
-        :returns: a list with a dictionary for each file
-        """
-        try:
-            root = ElementTree.fromstring(metalink_str)
-        except Exception as error:
-            self.logger.debug(metalink_str)
-            raise error
-        files = []
-
-        # metalink namespace
-        ns = '{urn:ietf:params:xml:ns:metalink}'
-
-        # loop over all <file> tags of the metalink string
-        for file_ml in root.findall(ns + 'file'):
-            # search for identity-tag
-            cur_did = file_ml.find(ns + 'identity')
-            if not ElementTree.iselement(cur_did):
-                raise RucioException('Failed to locate identity-tag inside %s' % ElementTree.tostring(file_ml))
-
-            # try extracting scope,name
-            scope, name = self._split_did_str(cur_did.text)
-
-            cur_file = {'scope': scope,
-                        'name': name,
-                        'bytes': None,
-                        'adler32': None,
-                        'md5': None,
-                        'sources': []}
-
-            size = file_ml.find(ns + 'size')
-            if ElementTree.iselement(size):
-                cur_file['bytes'] = int(size.text)
-
-            for cur_hash in file_ml.findall(ns + 'hash'):
-                hash_type = cur_hash.get('type')
-                if hash_type:
-                    cur_file[hash_type] = cur_hash.text
-
-            for rse_ml in file_ml.findall(ns + 'url'):
-                # check if location attrib (rse name) is given
-                rse = rse_ml.get('location')
-                pfn = rse_ml.text
-                cur_file['sources'].append({'pfn': pfn, 'rse': rse})
-
-            files.append(cur_file)
-
-        return files
-
     def _prepare_dest_dir(self, base_dir, dest_dir_name, file_name, no_subdir):
         """
         Builds the final destination path for a file and:
diff --git a/lib/rucio/common/exception.py b/lib/rucio/common/exception.py
--- a/lib/rucio/common/exception.py
+++ b/lib/rucio/common/exception.py
@@ -819,7 +819,7 @@ class MissingModuleException(RucioException):
     RucioException
     """
     def __init__(self, *args, **kwargs):
-        super(MissingModuleException, self).__init__(args, kwargs)
+        super(MissingModuleException, self).__init__(*args, **kwargs)
         self._message = "The module is not installed."
         self.error_code = 77
 
@@ -829,7 +829,7 @@ class ServerConnectionException(RucioException):
     RucioException
     """
     def __init__(self, *args, **kwargs):
-        super(ServerConnectionException, self).__init__(args, kwargs)
+        super(ServerConnectionException, self).__init__(*args, **kwargs)
         self._message = "Cannot connect to the Rucio server."
         self.error_code = 78
 
@@ -839,7 +839,7 @@ class NoFilesUploaded(RucioException):
     RucioException
     """
     def __init__(self, *args, **kwargs):
-        super(NoFilesUploaded, self).__init__(args, kwargs)
+        super(NoFilesUploaded, self).__init__(*args, **kwargs)
         self._message = "None of the given files have been uploaded."
         self.error_code = 79
 
@@ -849,7 +849,7 @@ class NotAllFilesUploaded(RucioException):
     RucioException
     """
     def __init__(self, *args, **kwargs):
-        super(NotAllFilesUploaded, self).__init__(args, kwargs)
+        super(NotAllFilesUploaded, self).__init__(*args, **kwargs)
         self._message = "Not all of the given files have been uploaded."
         self.error_code = 80
 
@@ -869,7 +869,7 @@ class UndefinedPolicy(RucioException):
     Cannot find a defined policy in the Rucio config
     """
     def __init__(self, *args, **kwargs):
-        super(UndefinedPolicy, self).__init__(args, kwargs)
+        super(UndefinedPolicy, self).__init__(*args, **kwargs)
         self._message = "No policy is defined."
         self.error_code = 82
 
@@ -879,7 +879,7 @@ class TransferToolTimeout(RucioException):
     Timeout from the transfer tool
     """
     def __init__(self, *args, **kwargs):
-        super(TransferToolTimeout, self).__init__(args, kwargs)
+        super(TransferToolTimeout, self).__init__(*args, **kwargs)
         self._message = "Timeout from the transfer tool."
         self.error_code = 83
 
@@ -889,7 +889,7 @@ class TransferToolWrongAnswer(RucioException):
     Wrong answer returned by the transfer tool
     """
     def __init__(self, *args, **kwargs):
-        super(TransferToolWrongAnswer, self).__init__(args, kwargs)
+        super(TransferToolWrongAnswer, self).__init__(*args, **kwargs)
         self._message = "Wrong answer returned by the transfer tool."
         self.error_code = 84
 
@@ -912,3 +912,13 @@ def __init__(self, *args, **kwargs):
         super(UnsupportedKeyType, self).__init__(*args, **kwargs)
         self._message = "Unsupported type for the key."
         self.error_code = 86
+
+
+class MetalinkJsonParsingError(RucioException):
+    """
+    Failed to parse input with metalink and json
+    """
+    def __init__(self, data, metalink_err, json_err, *args, **kwargs):
+        super(MetalinkJsonParsingError, self).__init__(*args, **kwargs)
+        self._message = 'Failed parsing of %s. MetalinkError: %s. JsonError: %s' % (data, metalink_err, json_err)
+        self.error_code = 87
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -48,6 +48,9 @@
 from logging.handlers import RotatingFileHandler
 from uuid import uuid4 as uuid
 from six import string_types
+from xml.etree import ElementTree
+
+from rucio.common.exception import InputValidationError, MetalinkJsonParsingError
 
 try:
     # Python 2
@@ -814,3 +817,101 @@ def parse_did_filter_from_string(input_string):
                     value = '0'
                 filters[key] = value
     return filters, type
+
+
+def parse_replicas_from_file(path):
+    """
+    Parses the output of list_replicas from a json or metalink file
+    into a dictionary. Metalink parsing is tried first and if it fails
+    it tries to parse json.
+
+    :param path: the path to the input file
+
+    :returns: a list with a dictionary for each file
+    """
+    with open(path) as fp:
+        try:
+            root = ElementTree.parse(fp).getroot()
+            return parse_replicas_metalink(root)
+        except ElementTree.ParseError as xml_err:
+            try:
+                return json.load(fp)
+            except ValueError as json_err:
+                raise MetalinkJsonParsingError(path, xml_err, json_err)
+
+
+def parse_replicas_from_string(string):
+    """
+    Parses the output of list_replicas from a json or metalink string
+    into a dictionary. Metalink parsing is tried first and if it fails
+    it tries to parse json.
+
+    :param string: the string to parse
+
+    :returns: a list with a dictionary for each file
+    """
+    try:
+        root = ElementTree.fromstring(string)
+        return parse_replicas_metalink(root)
+    except ElementTree.ParseError as xml_err:
+        try:
+            return json.loads(string)
+        except ValueError as json_err:
+            raise MetalinkJsonParsingError(string, xml_err, json_err)
+
+
+def parse_replicas_metalink(root):
+    """
+    Transforms the metalink tree into a list of dictionaries where
+    each dictionary describes a file with its replicas.
+    Will be called by parse_replicas_from_file and parse_replicas_from_string.
+
+    :param root: root node of the metalink tree
+
+    :returns: a list with a dictionary for each file
+    """
+    files = []
+
+    # metalink namespace
+    ns = '{urn:ietf:params:xml:ns:metalink}'
+    str_to_bool = {'true': True, 'True': True, 'false': False, 'False': False}
+
+    # loop over all <file> tags of the metalink string
+    for file_tag_obj in root.findall(ns + 'file'):
+        # search for identity-tag
+        identity_tag_obj = file_tag_obj.find(ns + 'identity')
+        if not ElementTree.iselement(identity_tag_obj):
+            raise InputValidationError('Failed to locate identity-tag inside %s' % ElementTree.tostring(file_tag_obj))
+
+        cur_file = {'did': identity_tag_obj.text,
+                    'adler32': None,
+                    'md5': None,
+                    'sources': []}
+
+        parent_dids = set()
+        parent_dids_tag_obj = file_tag_obj.find(ns + 'parents')
+        if ElementTree.iselement(parent_dids_tag_obj):
+            for did_tag_obj in parent_dids_tag_obj.findall(ns + 'did'):
+                parent_dids.add(did_tag_obj.text)
+        cur_file['parent_dids'] = parent_dids
+
+        size_tag_obj = file_tag_obj.find(ns + 'size')
+        cur_file['bytes'] = int(size_tag_obj.text) if ElementTree.iselement(size_tag_obj) else None
+
+        for hash_tag_obj in file_tag_obj.findall(ns + 'hash'):
+            hash_type = hash_tag_obj.get('type')
+            if hash_type:
+                cur_file[hash_type] = hash_tag_obj.text
+
+        for url_tag_obj in file_tag_obj.findall(ns + 'url'):
+            key_rename_map = {'location': 'rse'}
+            src = {}
+            for k, v in url_tag_obj.items():
+                k = key_rename_map.get(k, k)
+                src[k] = str_to_bool.get(v, v)
+            src['pfn'] = url_tag_obj.text
+            cur_file['sources'].append(src)
+
+        files.append(cur_file)
+
+    return files


Test patch: