Repository: gwastro/pycbc

Issue:
Conda installed version pycbc_submit_dax unable to start datafind connection
This error was found during running of pycbc_submit_dax, but can be reduced to the following:

```
python -c "from pycbc.frame import datafind_connection; server=datafind_connection()"
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/paul.hopkins/.conda/envs/pycbc_only/lib/python3.7/site-packages/pycbc/frame/frame.py", line 289, in datafind_connection
    host=server, port=port, cert_file=cert_file, key_file=key_file)
  File "/home/paul.hopkins/.conda/envs/pycbc_only/lib/python3.7/site-packages/glue/datafind.py", line 414, in __init__
    http_client.HTTPSConnection.__init__(self, host, **kwargs)
  File "/home/paul.hopkins/.conda/envs/pycbc_only/lib/python3.7/http/client.py", line 1358, in __init__
    blocksize=blocksize)
TypeError: __init__() takes from 1 to 2 positional arguments but 5 were given
```

Also this shows lots of errors from basename, which I did not see before.


Discussion:
@paulhopkins Do you have a reproducible command that produces this?

Also, while testing python3 is really useful for us, my suspicion is that you are trying to check if ARCCA can run production PyCBC jobs, which currently *must* happen on python2 (and from an environment that does not get updated, so that users in 2 years time can see what code versions were used directly).

So I'd definitely suggest dropping back to python2 for this testing. If CVMFS doesn't work for ARCCA, we should also consider how to create source-able environments for the current PyCBC releases (it is likely that GRB and all-sky will use different releases at any one time).
@spxiwh This is definitely a Python3 problem. Using Conda it is possible to install PyCBC with Python3, so you can either use this environment:

```
source /cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/bin/activate ligo-py36
```

or a self installed Conda environments: `conda install -n pycbc_only pycbc`. Although PyCBC is officially Python2 only, PyCBC still provides a lot of useful functionality to other users, so I wouldn't like to see it removed completely. I am just reporting errors until you have finished making it compatible with Python3.

Regarding ARCCA: CVMFS is working but it is not possible to use the supplied virtualenv straight out of the box on bare node. As you say, using the Singularity environment can get around this. On the other-hand Conda is able to provide a complete environment, and is already being used at ARCCA to supply PyCBC with a complete lscsoft Python2.7 environment. I am merely testing what is possible already.
@paulhopkins Okay great. I just want to distinguish between things that need to be fixed to be able to run production offline runs on ARCCA vs. things that can be fixed some months down the line. I did manage to use `pycbc_submit_dax` fine when I was testing a workflow in python3 some months back. Either something's gone backwards or there's some difference between ARCCA and hypatia in that regard .... Anyway that's some future Ian's problem!


To help future Ian with debugging here is the full stack trace and version information:

```
$ conda list pycbc
# packages in environment at /cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36:
#
# Name                    Version                   Build  Channel
pycbc                     1.14.1           py36h516909a_0    conda-forge
```

```
2019-09-10 08:45:13,892:INFO : Setting up connection to datafind server.
Traceback (most recent call last):
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/bin/pycbc_make_coinc_search_workflow", line 131, in <module>
    seg_file=science_seg_file)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/pycbc/workflow/datafind.py", line 144, in setup_datafind_workflow
    outputDir, tags=tags)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/pycbc/workflow/datafind.py", line 562, in setup_datafind_runtime_frames_single_call_perifo
    outputDir, tags=tags)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/pycbc/workflow/datafind.py", line 486, in setup_datafind_runtime_cache_single_call_perifo
    connection = setup_datafind_server_connection(cp, tags=tags)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/pycbc/workflow/datafind.py", line 832, in setup_datafind_server_connection
    return datafind_connection(datafind_server)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/pycbc/frame/frame.py", line 289, in datafind_connection
    host=server, port=port, cert_file=cert_file, key_file=key_file)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/site-packages/glue/datafind.py", line 414, in __init__
    http_client.HTTPSConnection.__init__(self, host, **kwargs)
  File "/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/envs/ligo-py36/lib/python3.6/http/client.py", line 1367, in __init__
    source_address)
TypeError: __init__() takes from 1 to 2 positional arguments but 5 were given
```

Okay thanks, this is very helpful .... This actually looks like a bug in `glue.datafind`.

Resolving patch:
diff --git a/pycbc/frame/frame.py b/pycbc/frame/frame.py
--- a/pycbc/frame/frame.py
+++ b/pycbc/frame/frame.py
@@ -21,7 +21,8 @@
 import lal
 import numpy
 import os.path, glob, time
-import glue.datafind
+import gwdatafind
+from six.moves.urllib.parse import urlparse
 from pycbc.types import TimeSeries, zeros
 
 
@@ -259,38 +260,7 @@ def datafind_connection(server=None):
     connection
         The open connection to the datafind server.
     """
-
-    if server:
-        datafind_server = server
-    else:
-        # Get the server name from the environment
-        if 'LIGO_DATAFIND_SERVER' in os.environ:
-            datafind_server = os.environ["LIGO_DATAFIND_SERVER"]
-        else:
-            err = "Trying to obtain the ligo datafind server url from "
-            err += "the environment, ${LIGO_DATAFIND_SERVER}, but that "
-            err += "variable is not populated."
-            raise ValueError(err)
-
-    # verify authentication options
-    if not datafind_server.endswith("80"):
-        cert_file, key_file = glue.datafind.find_credential()
-    else:
-        cert_file, key_file = None, None
-
-    # Is a port specified in the server URL
-    dfs_fields = datafind_server.split(':', 1)
-    server = dfs_fields[0]
-    port = int(dfs_fields[1]) if len(dfs_fields) == 2 else None
-
-    # Open connection to the datafind server
-    if cert_file and key_file:
-        connection = glue.datafind.GWDataFindHTTPSConnection(
-                host=server, port=port, cert_file=cert_file, key_file=key_file)
-    else:
-        connection = glue.datafind.GWDataFindHTTPConnection(
-                host=server, port=port)
-    return connection
+    return gwdatafind.connect(host=server)
 
 def frame_paths(frame_type, start_time, end_time, server=None, url_type='file'):
     """Return the paths to a span of frame files
@@ -324,8 +294,7 @@ def frame_paths(frame_type, start_time, end_time, server=None, url_type='file'):
     connection.find_times(site, frame_type,
                           gpsstart=start_time, gpsend=end_time)
     cache = connection.find_frame_urls(site, frame_type, start_time, end_time,urltype=url_type)
-    paths = [entry.path for entry in cache]
-    return paths
+    return [urlparse(entry).path for entry in cache]
 
 def query_and_read_frame(frame_type, channels, start_time, end_time,
                          sieve=None, check_integrity=False):
diff --git a/pycbc/workflow/datafind.py b/pycbc/workflow/datafind.py
--- a/pycbc/workflow/datafind.py
+++ b/pycbc/workflow/datafind.py
@@ -358,7 +358,7 @@ def setup_datafind_workflow(workflow, scienceSegs, outputDir, seg_file=None,
 def setup_datafind_runtime_cache_multi_calls_perifo(cp, scienceSegs,
                                                     outputDir, tags=None):
     """
-    This function uses the glue.datafind library to obtain the location of all
+    This function uses the `gwdatafind` library to obtain the location of all
     the frame files that will be needed to cover the analysis of the data
     given in scienceSegs. This function will not check if the returned frames
     cover the whole time requested, such sanity checks are done in the
@@ -438,7 +438,7 @@ def setup_datafind_runtime_cache_multi_calls_perifo(cp, scienceSegs,
 def setup_datafind_runtime_cache_single_call_perifo(cp, scienceSegs, outputDir,
                                               tags=None):
     """
-    This function uses the glue.datafind library to obtain the location of all
+    This function uses the `gwdatafind` library to obtain the location of all
     the frame files that will be needed to cover the analysis of the data
     given in scienceSegs. This function will not check if the returned frames
     cover the whole time requested, such sanity checks are done in the
@@ -518,7 +518,7 @@ def setup_datafind_runtime_cache_single_call_perifo(cp, scienceSegs, outputDir,
 def setup_datafind_runtime_frames_single_call_perifo(cp, scienceSegs,
                                               outputDir, tags=None):
     """
-    This function uses the glue.datafind library to obtain the location of all
+    This function uses the `gwdatafind` library to obtain the location of all
     the frame files that will be needed to cover the analysis of the data
     given in scienceSegs. This function will not check if the returned frames
     cover the whole time requested, such sanity checks are done in the
@@ -568,7 +568,7 @@ def setup_datafind_runtime_frames_single_call_perifo(cp, scienceSegs,
 def setup_datafind_runtime_frames_multi_calls_perifo(cp, scienceSegs,
                                                      outputDir, tags=None):
     """
-    This function uses the glue.datafind library to obtain the location of all
+    This function uses the `gwdatafind` library to obtain the location of all
     the frame files that will be needed to cover the analysis of the data
     given in scienceSegs. This function will not check if the returned frames
     cover the whole time requested, such sanity checks are done in the
@@ -899,7 +899,7 @@ def run_datafind_instance(cp, outputDir, connection, observatory, frameType,
         commands for reproducing what is done in this function to this
         directory.
     connection : datafind connection object
-        Initialized through the glue.datafind module, this is the open
+        Initialized through the `gwdatafind` module, this is the open
         connection to the datafind server.
     observatory : string
         The observatory to query frames for. Ex. 'H', 'L' or 'V'.  NB: not
@@ -952,10 +952,12 @@ def run_datafind_instance(cp, outputDir, connection, observatory, frameType,
     # directory to check if this was expected.
     log_datafind_command(observatory, frameType, startTime, endTime,
                          os.path.join(outputDir,'logs'), **dfKwargs)
-    logging.info("Asking datafind server for frames.")
-    dfCache = connection.find_frame_urls(observatory, frameType,
-                                        startTime, endTime, **dfKwargs)
-    logging.info("Frames returned")
+    logging.debug("Asking datafind server for frames.")
+    dfCache = lal.Cache.from_urls(
+        connection.find_frame_urls(observatory, frameType,
+                                   startTime, endTime, **dfKwargs),
+    )
+    logging.debug("Frames returned")
     # workflow format output file
     cache_file = File(ifo, 'DATAFIND', seg, extension='lcf',
                       directory=outputDir, tags=tags)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -56,6 +56,7 @@
                       'six>=1.10.0',
                       'ligo-segments',
                       'tqdm',
+                      'gwdatafind',
                       ]
 
 def find_files(dirname, relpath=None):


Test patch: