internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spacetelescope/jwql

Issue:
Add a specific error when user is missing a config file?
As @DanielKuebi was setting up the JWQL package, he found that he didn't have the `config.json` file that is required to run the package.

I looked back into `utils.py`, and I'm thinking it might be wise to raise a specific error when a user doesn't have the `config.json` file, so it is clear when that is the problem. Maybe even the error message could encourage them to email `jwql@stsci.edu` to ask a team member for assistance? 

If anyone thinks this isn't a good idea for security reasons, please say so.


Discussion:
@laurenmarietta I think this is a great idea.  I was also thinking that maybe we should make it more clear in the installation instructions in `README` that the user must have this file.

Resolving patch:
diff --git a/jwql/utils/utils.py b/jwql/utils/utils.py
--- a/jwql/utils/utils.py
+++ b/jwql/utils/utils.py
@@ -183,7 +183,16 @@ def get_config():
     settings : dict
         A dictionary that holds the contents of the config file.
     """
-    with open(os.path.join(__location__, 'config.json'), 'r') as config_file:
+    config_file_location = os.path.join(__location__, 'config.json')
+
+    if not os.path.isfile(config_file_location):
+        raise FileNotFoundError('The JWQL package requires a configuration file (config.json) '
+                                'to be placed within the jwql/utils directory. '
+                                'This file is missing. Please read the relevant wiki page '
+                                '(https://github.com/spacetelescope/jwql/wiki/'
+                                'Config-file) for more information.')
+
+    with open(config_file_location, 'r') as config_file:
         settings = json.load(config_file)
 
     return settings


Test patch:
Repository: spacetelescope/jwql

Issue:
Login/logout should not redirect back to homepage
Currently, the web app redirects to the homepage when clicking the login/logout button, regardless of which page the user was currently on.  We should switch this so that the page redirects to the page the user was on. 


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/oauth.py b/jwql/website/apps/jwql/oauth.py
--- a/jwql/website/apps/jwql/oauth.py
+++ b/jwql/website/apps/jwql/oauth.py
@@ -48,6 +48,8 @@ def login(request):
 from jwql.utils.constants import MONITORS
 from jwql.utils.utils import get_base_url, get_config
 
+PREV_PAGE = ''
+
 
 def register_oauth():
     """Register the ``jwql`` application with the ``auth.mast``
@@ -118,7 +120,7 @@ def authorize(request):
     cookie_args['httponly'] = True
 
     # Set the cookie
-    response = redirect("/")
+    response = redirect(PREV_PAGE)
     response.set_cookie("ASB-AUTH", token["access_token"], **cookie_args)
 
     return response
@@ -243,7 +245,10 @@ def login(request, user):
     """
 
     # Redirect to oauth login
+    global PREV_PAGE
+    PREV_PAGE = request.META.get('HTTP_REFERER')
     redirect_uri = os.path.join(get_base_url(), 'authorize')
+
     return JWQL_OAUTH.mast_auth.authorize_redirect(request, redirect_uri)
 
 
@@ -266,7 +271,9 @@ def logout(request):
         Outgoing response sent to the webpage
     """
 
-    response = redirect("/")
+    global PREV_PAGE
+    PREV_PAGE = request.META.get('HTTP_REFERER')
+    response = redirect(PREV_PAGE)
     response.delete_cookie("ASB-AUTH")
 
     return response


Test patch:
Repository: spacetelescope/jwql

Issue:
Make JWQL pip installable
Currently our `jwql` package is only installable by cloning the repository and running `setup.py`.  It would be easier for users (and perhaps easier for us when distributing our code (#294)) if it were also uploaded to PyPI and `pip` installable.  


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -4,8 +4,10 @@
 
 VERSION = '0.20.0'
 
-AUTHORS = 'Matthew Bourque, Sara Ogaz, Joe Filippazzo, Bryan Hilbert, Misty Cracraft, '
-AUTHORS += 'Graham Kanarek, Johannes Sahlmann, Lauren Chambers, Catherine Martlin'
+AUTHORS = 'Matthew Bourque, Lauren Chambers, Misty Cracraft, Joe Filippazzo, Bryan Hilbert, '
+AUTHORS += 'Graham Kanarek, Catherine Martlin, Johannes Sahlmann'
+
+DESCRIPTION = 'The James Webb Space Telescope Quicklook Project'
 
 REQUIRES = [
     'astropy',
@@ -31,7 +33,7 @@
 setup(
     name='jwql',
     version=VERSION,
-    description='The JWST Quicklook Project',
+    description=DESCRIPTION,
     url='https://github.com/spacetelescope/jwql.git',
     author=AUTHORS,
     author_email='jwql@stsci.edu',


Test patch:
Repository: spacetelescope/jwql

Issue:
Add README to style_guide directory
We are starting to have a range of helpful documents in our `jwql/style_guide` directory - the general style guide. This is great!

I am thinking it would now be helpful to include a `README.md` file in there, so that any prospective user who looks there is met with some information about what resources are available.


Discussion:

Resolving patch:
diff --git a/jwql/utils/monitor_template.py b/jwql/utils/monitor_template.py
--- a/jwql/utils/monitor_template.py
+++ b/jwql/utils/monitor_template.py
@@ -46,7 +46,7 @@
 
     Any monitoring script written for ``jwql`` must adhere to the
     ``jwql`` style guide located at:
-    https://github.com/spacetelescope/jwql/blob/master/style_guide/style_guide.md
+    https://github.com/spacetelescope/jwql/blob/master/style_guide/README.md
 """
 
 import os


Test patch:
Repository: spacetelescope/jwql

Issue:
Import environment bokeh version in HTML
As described in #425, having the environment Bokeh version NOT match the HTML template Bokeh version can cause problems.

I wrote some tests #429 that will test and make sure this is always the case in the future, but we could be even smarter about this if we programmatically set the Bokeh version using a Django variable. We could even include it in `context_processors` so it happens behind the scenes.


Discussion:
As i mentioned in #429, it might also help us to remove some of the potentially unnecessary calls to import `bokeh` in the various HTML templates; it would be a few less lines of code to have to worry about in situations like these.  

Resolving patch:
diff --git a/jwql/website/apps/jwql/context_processors.py b/jwql/website/apps/jwql/context_processors.py
--- a/jwql/website/apps/jwql/context_processors.py
+++ b/jwql/website/apps/jwql/context_processors.py
@@ -23,6 +23,8 @@
     As such, it will automatically be executed upon each request.
 """
 
+import bokeh
+
 import jwql
 from jwql.utils.constants import JWST_INSTRUMENT_NAMES, MONITORS
 
@@ -50,5 +52,6 @@ def base_context(request, user):
     context['tools'] = MONITORS
     context['user'] = user
     context['version'] = jwql.__version__
+    context['bokeh_version'] = bokeh.__version__
 
     return context


Test patch:
Repository: spacetelescope/jwql

Issue:
API ReadTheDocs failing
It seems as though our webhook to ReadTheDocs is not currently updating the API docs:
[![Documentation Status](https://readthedocs.org/projects/jwql/badge/?version=latest)](https://jwql.readthedocs.io/en/latest/?badge=latest)

Actually, it looks like our doc builds have been failing for a month!
https://readthedocs.org/projects/jwql/builds/

There is an error message here that should be helpful for investigating: https://readthedocs.org/projects/jwql/builds/9177249/




Discussion:

Resolving patch:
diff --git a/jwql/utils/utils.py b/jwql/utils/utils.py
--- a/jwql/utils/utils.py
+++ b/jwql/utils/utils.py
@@ -417,7 +417,7 @@ def check_config_for_key(key):
     and that the entry is not empty
 
     Parameters
-    -------
+    ----------
     key : str
         The configuration file key to verify
     """


Test patch:
diff --git a/docs/source/tests.rst b/docs/source/tests.rst
--- a/docs/source/tests.rst
+++ b/docs/source/tests.rst
@@ -20,8 +20,8 @@ test_dark_monitor.py
     :members:
     :undoc-members:
 
-test_edb_interface.py
----------------------
+test_edb.py
+-----------
 .. automodule:: jwql.tests.test_edb
     :members:
     :undoc-members:
@@ -33,7 +33,7 @@ test_instrument_properties.py
     :undoc-members:
 
 test_loading_times.py
---------------------
+---------------------
 .. automodule:: jwql.tests.test_loading_times
     :members:
     :undoc-members:
Repository: spacetelescope/jwql

Issue:
Simplify conda environments 
Since `conda` will automatically determine which libraries are needed for dependencies (e.g. `numpy_base` is installed when installing `numpy`), we could probably use to trim down our conda environments to only those high-level packages that are used within our repo, and `conda` will figure out the rest.  It might also be a good time to make sure the `conda` environment is consistent with the dependencies listed in `setup.py` and `requirements.txt`


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -4,21 +4,26 @@
 
 VERSION = '0.21.0'
 
-AUTHORS = 'Matthew Bourque, Lauren Chambers, Misty Cracraft, Joe Filippazzo, Bryan Hilbert, '
-AUTHORS += 'Graham Kanarek, Catherine Martlin, Johannes Sahlmann'
+AUTHORS = 'Matthew Bourque, Misty Cracraft, Joe Filippazzo, Bryan Hilbert, '
+AUTHORS += 'Graham Kanarek, Catherine Martlin, Johannes Sahlmann, Ben Sunnquist'
 
 DESCRIPTION = 'The James Webb Space Telescope Quicklook Project'
 
 REQUIRES = [
+    'asdf>=2.3.3',
     'astropy>=3.2.1',
     'astroquery>=0.3.9',
     'authlib',
     'bokeh>=1.0',
+    'codecov',
     'django>=2.0',
+    'flake8',
+    'inflection',
+    'ipython',
     'jinja2',
     'jsonschema==2.6.0',
-    'jwedb',
-    'jwst',
+    'jwedb>=0.0.3',
+    'jwst==0.13.0',
     'matplotlib',
     'numpy',
     'numpydoc',
@@ -26,9 +31,12 @@
     'psycopg2',
     'pysiaf',
     'pytest',
+    'pytest-cov',
+    'scipy',
     'sphinx',
     'sqlalchemy',
-    'stsci_rtd_theme'
+    'stsci_rtd_theme',
+    'twine'
 ]
 
 setup(


Test patch:
Repository: spacetelescope/jwql

Issue:
Avoid using astroconda-dev
As we approach our 1.0 release, we should not be relying on `astroconda-dev` for some of our dependencies, since that channel is for development purposes and may break from time to time.  We should go through our list of dependencies and make sure we can install them from default `conda` channels, `astroconda`, or `pip`. 


Discussion:

Resolving patch:
diff --git a/jwql/instrument_monitors/miri_monitors/data_trending/dt_cron_job.py b/jwql/instrument_monitors/miri_monitors/data_trending/dt_cron_job.py
--- a/jwql/instrument_monitors/miri_monitors/data_trending/dt_cron_job.py
+++ b/jwql/instrument_monitors/miri_monitors/data_trending/dt_cron_job.py
@@ -19,9 +19,9 @@
 ----------
 
 '''
-import .utils.mnemonics as mn
-import .utils.sql_interface as sql
-from .utils.process_data import whole_day_routine, wheelpos_routine
+import utils.mnemonics as mn
+import utils.sql_interface as sql
+from utils.process_data import whole_day_routine, wheelpos_routine
 from jwql.utils.engineering_database import query_single_mnemonic
 
 import pandas as pd
diff --git a/jwql/instrument_monitors/nirspec_monitors/data_trending/dt_cron_job.py b/jwql/instrument_monitors/nirspec_monitors/data_trending/dt_cron_job.py
--- a/jwql/instrument_monitors/nirspec_monitors/data_trending/dt_cron_job.py
+++ b/jwql/instrument_monitors/nirspec_monitors/data_trending/dt_cron_job.py
@@ -24,9 +24,9 @@
 Notes
 -----
 '''
-import .utils.mnemonics as mn
-import .utils.sql_interface as sql
-from .utils.process_data import whole_day_routine, wheelpos_routine
+import utils.mnemonics as mn
+import utils.sql_interface as sql
+from utils.process_data import whole_day_routine, wheelpos_routine
 from jwql.utils.engineering_database import query_single_mnemonic
 
 import pandas as pd
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -23,7 +23,6 @@
     'jinja2',
     'jsonschema==2.6.0',
     'jwedb>=0.0.3',
-    'jwst==0.13.0',
     'matplotlib',
     'numpy',
     'numpydoc',
@@ -39,6 +38,8 @@
     'twine'
 ]
 
+DEPENDENCY_LINKS = ['git+https://github.com/spacetelescope/jwst#0.13.0']
+
 setup(
     name='jwql',
     version=VERSION,
@@ -51,6 +52,7 @@
     classifiers=['Programming Language :: Python'],
     packages=find_packages(),
     install_requires=REQUIRES,
+    dependency_links=DEPENDENCY_LINKS,
     include_package_data=True,
     include_dirs=[np.get_include()],
 )


Test patch:
Repository: spacetelescope/jwql

Issue:
DEPENDENCY_LINKS in setup.py causing bug in logging_functions
With the introduction of the `DEPENDENCY_LINKS` variable in `setup.py`, the logging of monitors is now failing to log the versions of depenencies listed, since the `REQUIRES` is not immediately followed by `setup()`:

```python
for i, line in enumerate(data):
            if 'REQUIRES = [' in line:
                begin = i + 1
            elif 'setup(' in line:
                end = i - 2
```

The solution is so simple move `DEPENDENCY _LINKS` to be defined before `REQUIRES`.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,6 +9,7 @@
 
 DESCRIPTION = 'The James Webb Space Telescope Quicklook Project'
 
+DEPENDENCY_LINKS = ['git+https://github.com/spacetelescope/jwst#0.13.0']
 REQUIRES = [
     'asdf>=2.3.3',
     'astropy>=3.2.1',
@@ -38,8 +39,6 @@
     'twine'
 ]
 
-DEPENDENCY_LINKS = ['git+https://github.com/spacetelescope/jwst#0.13.0']
-
 setup(
     name='jwql',
     version=VERSION,


Test patch:
Repository: spacetelescope/jwql

Issue:
Add a 'conda env export' to logging module
We could add the output of `conda env export` to the beginning of log files so that the exact environment being used to run a script is logged. 


Discussion:

Resolving patch:
diff --git a/jwql/utils/logging_functions.py b/jwql/utils/logging_functions.py
--- a/jwql/utils/logging_functions.py
+++ b/jwql/utils/logging_functions.py
@@ -10,8 +10,9 @@
 -------
 
     - Catherine Martlin
-    - Alex Viana (WFC3 QL Version)
+    - Alex Viana (wfc3ql Version)
     - Matthew Bourque
+    - Jason Neal
 
 Use
 ---
@@ -59,6 +60,7 @@ def my_main_function():
 import os
 import pwd
 import socket
+import subprocess
 import sys
 import time
 import traceback
@@ -236,7 +238,10 @@ def wrapped(*args, **kwargs):
             except (ImportError, AttributeError) as err:
                 logging.warning(err)
 
-        logging.info('')
+        environment = subprocess.check_output(['conda', 'env', 'export'], universal_newlines=True)
+        logging.info('Environment:')
+        for line in environment.split('\n'):
+            logging.info(line)
 
         # Call the function and time it
         t1_cpu = time.clock()


Test patch:
Repository: spacetelescope/jwql

Issue:
Cron jobs for monitors currently failing 
Traceback (most recent call last):
  File "/home/jwqladm/repositories/jwql/jwql/instrument_monitors/common_monitors/bias_monitor.py", line 58, in <module>
    from jwql.instrument_monitors.common_monitors.dark_monitor import mast_query_darks
  File "/home/jwqladm/repositories/jwql/jwql/instrument_monitors/common_monitors/dark_monitor.py", line 77, in <module>
    from jwql.jwql_monitors import monitor_mast
  File "/home/jwqladm/repositories/jwql/jwql/jwql_monitors/monitor_mast.py", line 25, in <module>
    from bokeh.embed import components
  File "/grp/jwst/ins/jwql/envs/miniconda3/envs/jwql-3.6/lib/python3.6/site-packages/bokeh/__init__.py", line 81, in <module>
    from .util import logconfig
  File "/grp/jwst/ins/jwql/envs/miniconda3/envs/jwql-3.6/lib/python3.6/site-packages/bokeh/util/logconfig.py", line 87, in <module>
    level = settings.py_log_level()
  File "/grp/jwst/ins/jwql/envs/miniconda3/envs/jwql-3.6/lib/python3.6/site-packages/bokeh/settings.py", line 310, in __call__
    return self._convert(os.environ[self._env_var])
  File "/grp/jwst/ins/jwql/envs/miniconda3/envs/jwql-3.6/lib/python3.6/site-packages/bokeh/settings.py", line 236, in convert_logging
    raise ValueError("Cannot convert {} to log level, valid values are: {}".format(value, ", ".join(_log_levels)))
ValueError: Cannot convert WARN to log level, valid values are: CRITICAL, ERROR, WARNING, INFO, DEBUG, TRACE, NONE


Discussion:
This appears to be a `bokeh` version issue that introduced itself in version `1.4`. Reverting back to `1.3` appears to work. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -15,7 +15,7 @@
     'astropy>=3.2.1',
     'astroquery>=0.3.9',
     'authlib',
-    'bokeh>=1.0',
+    'bokeh>=1.0,<1.4',
     'codecov',
     'django>=2.0',
     'flake8',


Test patch:
Repository: spacetelescope/jwql

Issue:
Switch EDB to use MAST token from config.json always
Currently the EDB feature of the JWQL web app attempts to authenticate users with whichever MAST token is currently cached (https://github.com/spacetelescope/jwql/blob/develop/jwql/utils/credentials.py#L45), and if that doesn't succeed, _then_ it uses the `mast_token` key in the `config.json` file.  This seems problematic if users are creating new tokens but then attempting to perform EDB queries in the same browser.  We should probably switch this to just use the `mast_token` key in `config.json` file always. 


Discussion:

Resolving patch:
diff --git a/jwql/utils/credentials.py b/jwql/utils/credentials.py
--- a/jwql/utils/credentials.py
+++ b/jwql/utils/credentials.py
@@ -41,11 +41,6 @@ def get_mast_token(request=None):
         print('Authenticated with Astroquery MAST magic')
         return None
     else:
-        if request is not None:
-            token = str(request.POST.get('access_token'))
-            if token != 'None':
-                print('Authenticated with cached MAST token.')
-                return token
         try:
             # check if token is available via config file
             check_config_for_key('mast_token')


Test patch:
Repository: spacetelescope/jwql

Issue:
Fix DataTables warning when viewing headers in web app
When viewing a header for a file in the web app, we encounter this error:

<img width="444" alt="Screen Shot 2020-01-28 at 12 08 27" src="https://user-images.githubusercontent.com/2250769/73287273-023beb00-41c7-11ea-8b99-31d0ff760da4.png">


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/data_containers.py b/jwql/website/apps/jwql/data_containers.py
--- a/jwql/website/apps/jwql/data_containers.py
+++ b/jwql/website/apps/jwql/data_containers.py
@@ -576,12 +576,13 @@ def get_header_info(filename):
 
     # Extract header information from file
     for ext in range(0, len(hdulist)):
+    #for ext in range(0, 1):
 
         # Initialize dictionary to store header information for particular extension
         header_info[ext] = {}
 
         # Get header
-        header = fits.getheader(fits_filepath, ext=ext)
+        header = hdulist[ext].header
 
         # Determine the extension name
         if ext == 0:
@@ -601,11 +602,12 @@ def get_header_info(filename):
 
     # Build tables
     for ext in header_info:
-        table = Table([header_info[ext]['keywords'], header_info[ext]['values']], names=('Key', 'Value'))
-        temp_path_for_html = os.path.join(tempfile.mkdtemp(), '{}_table.html'.format(header_info[ext]['EXTNAME']))
-        with open(temp_path_for_html, 'w') as f:
-            table.write(f, format='jsviewer', jskwargs={'display_length': 20})
-        header_info[ext]['table'] = open(temp_path_for_html, 'r').read()
+        data_dict = {}
+        data_dict['Keyword'] = header_info[ext]['keywords']
+        data_dict['Value'] = header_info[ext]['values']
+        header_info[ext]['table'] = pd.DataFrame(data_dict)
+        header_info[ext]['table_rows'] = header_info[ext]['table'].values
+        header_info[ext]['table_columns'] = header_info[ext]['table'].columns.values
 
     return header_info
 
@@ -1153,11 +1155,12 @@ def thumbnails_ajax(inst, proposal=None):
         data_dict['file_data'][rootname] = {}
         data_dict['file_data'][rootname]['filename_dict'] = filename_dict
         data_dict['file_data'][rootname]['available_files'] = available_files
+        data_dict['file_data'][rootname]['suffixes'] = [filename_parser(filename)['suffix'] for filename in available_files]
         try:
             data_dict['file_data'][rootname]['expstart'] = get_expstart(inst, rootname)
             data_dict['file_data'][rootname]['expstart_iso'] = Time(data_dict['file_data'][rootname]['expstart'], format='mjd').iso.split('.')[0]
         except:
-            print("issue with get_expstart... for {}".format(rootname))
+            print("issue with get_expstart for {}".format(rootname))
 
     # Extract information for sorting with dropdown menus
     # (Don't include the proposal as a sorting parameter if the proposal has already been specified)


Test patch:
Repository: spacetelescope/jwql

Issue:
Update Bokeh to latest version
I remember there was some reason that we were holding off on upgrading Bokeh from 1.3.4. However, Bokeh is now up to version 2.2.1 I believe. We should look into upgrading the version used for JWQL in order to take advantage of new features and so that we minimize the number of plots created under 1.3.4 which may need to be tweaked to work under the new version.

For example, one difference I ran into today was that the keyword "legend", which is used in 1.3.4 to denote the string printed in the legend for a particular element, has been changed to "legend_label" in version 2.2.1.


Discussion:
Good idea, let's try to get to this next sprint. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -16,7 +16,7 @@
     'astropy>=3.2.1',
     'astroquery>=0.3.9',
     'authlib',
-    'bokeh>=1.0,<1.4',
+    'bokeh',
     'codecov',
     'crds',
     'cryptography',


Test patch:
Repository: spacetelescope/jwql

Issue:
Upgrade Django to 3.0
Django 3.0 is out, and since it is a major release, we should consider upgrading to this.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -20,7 +20,7 @@
     'codecov',
     'crds',
     'cryptography',
-    'django>=2.0,<3.0',
+    'django',
     'flake8',
     'inflection',
     'ipython',


Test patch: