internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spacetelescope/jwql

Issue:
Bypass web app login for local developers
Currently the web app log in button only works through `localhost` for myself, and will likely stay that way based on how things are set up on the MAST side of things.  We should implement a workaround that bypasses this login when running the web app locally.


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/oauth.py b/jwql/website/apps/jwql/oauth.py
--- a/jwql/website/apps/jwql/oauth.py
+++ b/jwql/website/apps/jwql/oauth.py
@@ -207,7 +207,8 @@ def auth_required(fn):
     @auth_info
     def check_auth(request, user, **kwargs):
         """Check if the user is authenticated through ``auth.mast``.
-        If not, perform the authorization.
+        If not, perform the authorization, unless the user is running
+        the web app locally, in which case authorization is bypassed.
 
         Parameters
         ----------
@@ -222,15 +223,25 @@ def check_auth(request, user, **kwargs):
             The decorated function
         """
 
-        # If user is currently anonymous, require a login
-        if user['ezid']:
+        # Determine domain
+        base_url = get_base_url()
+        if '127.0.0.1' in base_url:
+            domain = '127.0.0.1'
+        else:
+            domain = base_url.split('//')[-1]
+
+        # If running web app locally, the user does not need to be authenticated
+        if domain == '127.0.0.1':
+            return fn(request, user, **kwargs)
 
+        # If the user has an ezid, then they are already authenticated
+        elif user['ezid']:
             return fn(request, user, **kwargs)
 
+        # Otherwise, the user is not authenticated and can't continue
         else:
             template = 'not_authenticated.html'
             context = {'inst': ''}
-
             return render(request, template, context)
 
     return check_auth
diff --git a/jwql/website/apps/jwql/views.py b/jwql/website/apps/jwql/views.py
--- a/jwql/website/apps/jwql/views.py
+++ b/jwql/website/apps/jwql/views.py
@@ -760,6 +760,12 @@ def view_image(request, user, inst, file_root, rewrite=False):
     # Create a form instance
     form = InstrumentAnomalySubmitForm(request.POST or None, instrument=inst.lower(), initial={'anomaly_choices': current_anomalies})
 
+    # If user is running the web app locally and has not authenticated,
+    # then replace ezid with 'dev'
+    if '127.0.0.1' in get_base_url():
+        if user['ezid'] is None:
+            user['ezid'] = 'dev'
+
     # If this is a POST request, process the form data
     if request.method == 'POST':
         anomaly_choices = dict(request.POST)['anomaly_choices']


Test patch:
Repository: spacetelescope/jwql

Issue:
generate_preview_images needs not to make so many thumbnails
Currently the `generate_preview_images.py` module, which creates preview images and thumbnails for all datasets in the filesystem, creates thumbnails for all integrations of an observation.  In some observations, this number can be in the thousands, resulting in thousands of thumbnails (e.g. `jw00601/jw00601012001_gs-fg_2018221005716_uncal_integ1000`).  Since the JWQL app only uses thumbnails to serve as buttons to the `view_image` webpages, having thumbnails for more than one integration is unnecessary.

This module should be modified to only create thumbnails for one integration, when applicable.  


Discussion:

Resolving patch:
diff --git a/jwql/utils/preview_image.py b/jwql/utils/preview_image.py
--- a/jwql/utils/preview_image.py
+++ b/jwql/utils/preview_image.py
@@ -195,7 +195,7 @@ def get_data(self, filename, ext):
                 for exten in hdulist:
                     try:
                         extnames.append(exten.header['EXTNAME'])
-                    except:
+                    except KeyError:
                         pass
                 if ext in extnames:
                     dimensions = len(hdulist[ext].data.shape)
@@ -389,16 +389,18 @@ def make_image(self, max_img_size=8):
             self.save_image(outfile, thumbnail=False)
             plt.close()
 
-            # Create thumbnail image matplotlib object
-            if self.thumbnail_output_directory is None:
-                outdir = indir
-            else:
-                outdir = self.thumbnail_output_directory
-            outfile = os.path.join(outdir, infile.split('.')[0] + suffix)
-            self.make_figure(frame, i, minval, maxval, self.scaling.lower(),
-                             maxsize=max_img_size, thumbnail=True)
-            self.save_image(outfile, thumbnail=True)
-            plt.close()
+            # Create thumbnail image matplotlib object, only for the
+            # first integration
+            if i == 0:
+                if self.thumbnail_output_directory is None:
+                    outdir = indir
+                else:
+                    outdir = self.thumbnail_output_directory
+                outfile = os.path.join(outdir, infile.split('.')[0] + suffix)
+                self.make_figure(frame, i, minval, maxval, self.scaling.lower(),
+                                 maxsize=max_img_size, thumbnail=True)
+                self.save_image(outfile, thumbnail=True)
+                plt.close()
 
     def save_image(self, fname, thumbnail=False):
         """


Test patch:
Repository: spacetelescope/jwql

Issue:
Update cross-origin request in test/ops web servers
Currently the test and ops servers do not render the instrument archive or proposal webpages.  I think it is related to this error that shows up on the web console when loading those pages:

```
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://<dev server>/ajax/FGS/archive/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)
```

I'm thinking that maybe there is something hard-coded in the web app code that is pinging the dev server for requests from the test/ops servers.  This should be fixed. 


Discussion:

Resolving patch:
diff --git a/jwql/utils/utils.py b/jwql/utils/utils.py
--- a/jwql/utils/utils.py
+++ b/jwql/utils/utils.py
@@ -508,7 +508,7 @@ def get_base_url():
 
     username = getpass.getuser()
     if username == get_config()['admin_account']:
-        base_url = 'https://dljwql.stsci.edu'
+        base_url = 'https://{}.stsci.edu'.format(get_config()['server_name'])
     else:
         base_url = 'http://127.0.0.1:8000'
 
diff --git a/jwql/website/apps/jwql/views.py b/jwql/website/apps/jwql/views.py
--- a/jwql/website/apps/jwql/views.py
+++ b/jwql/website/apps/jwql/views.py
@@ -72,9 +72,6 @@
 from .forms import FileSearchForm
 
 
-FILESYSTEM_DIR = os.path.join(get_config()['jwql_dir'], 'filesystem')
-
-
 def anomaly_query(request):
     """The anomaly query form page"""
 
@@ -282,14 +279,14 @@ def archived_proposals_ajax(request, inst):
     for filename in filenames_public:
         try:
             relative_filepath = filesystem_path(filename, check_existence=False)
-            full_filepath = os.path.join(FILESYSTEM_DIR, 'public', relative_filepath)
+            full_filepath = os.path.join(get_config()['filesystem'], 'public', relative_filepath)
             filenames.append(full_filepath)
         except ValueError:
             print('Unable to determine filepath for {}'.format(filename))
     for filename in filenames_proprietary:
         try:
             relative_filepath = filesystem_path(filename, check_existence=False)
-            full_filepath = os.path.join(FILESYSTEM_DIR, 'proprietary', relative_filepath)
+            full_filepath = os.path.join(get_config()['filesystem'], 'proprietary', relative_filepath)
             filenames.append(full_filepath)
         except ValueError:
             print('Unable to determine filepath for {}'.format(filename))


Test patch:
Repository: spacetelescope/jwql

Issue:
Add 'submit sucessful' message when submitting anomalies
Currently, when a user submits an anomaly on the `view_image` view, the page is refreshed and nothing else happens.  It would be nice to have some sort of success message pop up if the anomaly submission was successful so that the user knows that something actually happened. 


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/views.py b/jwql/website/apps/jwql/views.py
--- a/jwql/website/apps/jwql/views.py
+++ b/jwql/website/apps/jwql/views.py
@@ -43,6 +43,7 @@
 from bokeh.layouts import layout
 from bokeh.embed import components
 from django.http import HttpResponse, JsonResponse
+from django.contrib import messages
 from django.shortcuts import redirect, render
 
 from jwql.database.database_interface import load_connection
@@ -753,11 +754,14 @@ def view_image(request, inst, file_root, rewrite=False):
     # Create a form instance
     form = InstrumentAnomalySubmitForm(request.POST or None, instrument=inst.lower(), initial={'anomaly_choices': current_anomalies})
 
-    # If this is a POST request, process the form data
-    if request.method == 'POST':
+    # If this is a POST request and the form is filled out, process the form data
+    if request.method == 'POST' and 'anomaly_choices' in dict(request.POST):
         anomaly_choices = dict(request.POST)['anomaly_choices']
         if form.is_valid():
             form.update_anomaly_table(file_root, 'unknown', anomaly_choices)
+            messages.success(request, "Anomaly submitted successfully")
+        else:
+            messages.error(request, "Failed to submit anomaly")
 
     # Build the context
     context = {'inst': inst,
diff --git a/jwql/website/jwql_proj/settings.py b/jwql/website/jwql_proj/settings.py
--- a/jwql/website/jwql_proj/settings.py
+++ b/jwql/website/jwql_proj/settings.py
@@ -28,7 +28,7 @@
 import os
 
 from jwql.utils.utils import get_config
-
+from django.contrib.messages import constants as messages
 
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -75,6 +75,7 @@
             'extensions': ['jwql.website.jwql_proj.jinja2.DjangoNow'],
             'context_processors': [
                 'jwql.website.apps.jwql.context_processors.base_context',
+                'django.contrib.messages.context_processors.messages'
             ],
         },
     },
@@ -91,6 +92,14 @@
     },
 ]
 
+MESSAGE_TAGS = {
+        messages.DEBUG: 'alert-secondary',
+        messages.INFO: 'alert-info',
+        messages.SUCCESS: 'alert-success',
+        messages.WARNING: 'alert-warning',
+        messages.ERROR: 'alert-danger',
+ }
+
 WSGI_APPLICATION = 'jwql.website.jwql_proj.wsgi.application'
 
 


Test patch:
Repository: spacetelescope/jwql

Issue:
Add code to manage.py to create necessesary symlinks to run web app locally 
In order to run the JWQL web app locally, one must create symbolic links to the `outputs`, `thumbnails`, `preview_images`, and `filesystem` directories.  We can add some code in `website.manage.py` in order to do this automatically.  Something like this:


```python
from jwql.utils.utils import get_config()

if __name__ == "__main__":

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "jwql_proj.settings")

    # Create symbolic links here (if they don't already exist)
    for directory in ['filesystem', 'outputs', 'preview_image_filesystem', 'thumbnails_filesystem']:
        path = get_config()[directory]
        # code to create symlink

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)
```

Credit @york-stsci for the suggestion!


Discussion:

Resolving patch:
diff --git a/jwql/website/manage.py b/jwql/website/manage.py
--- a/jwql/website/manage.py
+++ b/jwql/website/manage.py
@@ -36,10 +36,25 @@
 import os
 import sys
 
+from jwql.utils.utils import get_config
+
 if __name__ == "__main__":
 
     os.environ.setdefault("DJANGO_SETTINGS_MODULE", "jwql_proj.settings")
 
+    directory_mapping = {
+        'filesystem': 'filesystem',
+        'outputs': 'outputs',
+        'preview_image_filesystem': 'preview_images',
+        'thumbnail_filesystem': 'thumbnails'
+        }
+
+    for directory in ['filesystem', 'outputs', 'preview_image_filesystem', 'thumbnail_filesystem']:
+        symlink_location = os.path.join(os.path.dirname(__file__), 'apps', 'jwql', 'static', directory_mapping[directory])
+        if not os.path.exists(symlink_location):
+            symlink_path = get_config()[directory]
+            os.symlink(symlink_path, symlink_location)
+
     try:
         from django.core.management import execute_from_command_line
     except ImportError as exc:


Test patch:
Repository: spacetelescope/jwql

Issue:
Add x1d to the list of ignored suffixes
x1d files are ascii tables, and so should be ignored. Currently x1d shows up as an option when looking at preview images: 
https://jwql.stsci.edu/NIRISS/jw01324001001_11101_00004_nis/

Switching to this shows a broken link, but also seems to resize things such that when switching back to a different suffix, the preview image is very small.




Discussion:

Resolving patch:
diff --git a/jwql/utils/constants.py b/jwql/utils/constants.py
--- a/jwql/utils/constants.py
+++ b/jwql/utils/constants.py
@@ -235,7 +235,7 @@
 GUIDER_SUFFIX_TYPES = ['stream', 'stacked_uncal', 'image_uncal', 'stacked_cal', 'image_cal']
 
 # JWQL should ignore some filetypes in the filesystem.
-IGNORED_SUFFIXES = ['original', 'stream']
+IGNORED_SUFFIXES = ['original', 'stream', 'x1d', 'x1dints', 'c1d']
 
 # Instrument monitor database tables
 INSTRUMENT_MONITOR_DATABASE_TABLES = {


Test patch:
Repository: spacetelescope/jwql

Issue:
Image view for segmented files is not working
e.g. https://jwql.stsci.edu/NIRCam/jw01442001001_04103_00001-seg004_nrca3/

This is giving a "Page not found" error.


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/urls.py b/jwql/website/apps/jwql/urls.py
--- a/jwql/website/apps/jwql/urls.py
+++ b/jwql/website/apps/jwql/urls.py
@@ -81,7 +81,7 @@
     re_path(r'^(?P<inst>({}))/$'.format(instruments), views.instrument, name='instrument'),
     re_path(r'^(?P<inst>({}))/archive/$'.format(instruments), views.archived_proposals, name='archive'),
     re_path(r'^(?P<inst>({}))/unlooked/$'.format(instruments), views.unlooked_images, name='unlooked'),
-    re_path(r'^(?P<inst>({}))/(?P<file_root>[\w]+)/$'.format(instruments), views.view_image, name='view_image'),
+    re_path(r'^(?P<inst>({}))/(?P<file_root>[\w-]+)/$'.format(instruments), views.view_image, name='view_image'),
     re_path(r'^(?P<inst>({}))/(?P<file_root>.+)_(?P<filetype>.+)/explore_image/'.format(instruments), views.explore_image, name='explore_image'),
     re_path(r'^(?P<inst>({}))/(?P<filename>.+)_(?P<filetype>.+)/header/'.format(instruments), views.view_header, name='view_header'),
     re_path(r'^(?P<inst>({}))/archive/(?P<proposal>[\d]{{1,5}})/obs(?P<observation>[\d]{{1,3}})/$'.format(instruments), views.archive_thumbnails_per_observation, name='archive_thumb_per_obs'),


Test patch:
Repository: spacetelescope/jwql

Issue:
Update the About page
Our list of contributors on the About page needs to be updated. In addition to removing those no longer on the team, we also need to add @BradleySappington and @penaguerrero 


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/data_containers.py b/jwql/website/apps/jwql/data_containers.py
--- a/jwql/website/apps/jwql/data_containers.py
+++ b/jwql/website/apps/jwql/data_containers.py
@@ -11,6 +11,7 @@
     - Lauren Chambers
     - Matthew Bourque
     - Teagan King
+    - Bryan Hilbert
 
 Use
 ---


Test patch:
Repository: spacetelescope/jwql

Issue:
Add new anomaly types for NIRCam
Discussed in the NIRCam meeting today:

Should we add claws and wisps to the list of NIRCam anomalies? 
@bsunnquist is the best one to make the call on this.

We should also add a "scattered light" entry, to capture exposures with unexpected scattered light features.


Discussion:
Yes, I think they should be added. The only issue is most claws are faint and require stacks of several exposures to see them - for these, I'll probably add some flagging option as part of my claw monitor page. But stronger claws will still show in single exposures, so I still think it's worth having it as an option.

Wisps will probably show in the majority of exposures on e.g. B4, but I still think it's worth tracking them as well, to help diagnose/model them.

Resolving patch:
diff --git a/jwql/utils/constants.py b/jwql/utils/constants.py
--- a/jwql/utils/constants.py
+++ b/jwql/utils/constants.py
@@ -80,7 +80,10 @@
     'row_pull_down': ['miri'],
     'LRS_Contamination': ['miri'],
     'tree_rings': ['miri'],
-    'scattered_light': ['niriss'],
+    'scattered_light': ['niriss', 'nircam'],
+    'claws': ['nircam'],
+    'wisps': ['nircam'],
+    'tilt_event': ['nircam'],
     'light_saber': ['niriss'],
     # additional anomalies:
     'other': ['fgs', 'miri', 'nircam', 'niriss', 'nirspec']}


Test patch:
Repository: spacetelescope/jwql

Issue:
Dark monitor page failing to load
The dark monitor page for all instruments currently crashes with the traceback: 

name 'xdim' is not defined
monitor_dark_bokeh.py, line 241, in create_plot


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/monitor_pages/monitor_dark_bokeh.py b/jwql/website/apps/jwql/monitor_pages/monitor_dark_bokeh.py
--- a/jwql/website/apps/jwql/monitor_pages/monitor_dark_bokeh.py
+++ b/jwql/website/apps/jwql/monitor_pages/monitor_dark_bokeh.py
@@ -238,9 +238,10 @@ def create_plot(self):
                 view = read_png(self.dark_image_picture)
 
                 # Display the 32-bit RGBA image
+                ydim, xdim = view.shape
                 dim = max(xdim, ydim)
                 self.plot = figure(x_range=(0, xdim), y_range=(0, ydim), tools='pan,box_zoom,reset,wheel_zoom,save')
-                self.plot.image_rgba(image=[img], x=0, y=0, dw=xdim, dh=ydim)
+                self.plot.image_rgba(image=[view], x=0, y=0, dw=xdim, dh=ydim)
                 self.plot.xaxis.visible = False
                 self.plot.yaxis.visible = False
 


Test patch:
Repository: spacetelescope/jwql

Issue:
Add missing subarrrays to constants
While searching for NIRCam data, I noticed that there are a number of subarrays that are missing from the query form. These values come from utils/constants.py, in the SUBARRAYS_PER_DETECTOR entry.

The full lists of possible subarrays for all the instruments are given in the schema for the stdatamodels:  https://github.com/spacetelescope/stdatamodels/blob/master/src/stdatamodels/jwst/datamodels/schemas/subarray.schema.yaml

We should update our lists in constants.py to match these lists. 


Discussion:

Resolving patch:
diff --git a/jwql/utils/constants.py b/jwql/utils/constants.py
--- a/jwql/utils/constants.py
+++ b/jwql/utils/constants.py
@@ -28,6 +28,7 @@
     ``utils.py``
 """
 
+import asdf
 import inflection
 
 # Each amplifier is represented by 2 tuples, the first for x coordinates
@@ -913,58 +914,14 @@ class QueryConfigKeys:
     "SUBGRISMSTRIPE256",
 ]
 
+schema = asdf.schema.load_schema("http://stsci.edu/schemas/jwst_datamodel/subarray.schema")
 SUBARRAYS_PER_INSTRUMENT = {
-    "nircam": [
-        "FULL",
-        "FULLP",
-        "SUB640",
-        "SUB320",
-        "SUB160",
-        "SUB400P",
-        "SUB160P",
-        "SUB64P",
-        "SUB32TATS",
-        "SUB640A210R",
-        "SUB640ASWB",
-        "SUB320A335R",
-        "SUB320A430R",
-        "SUB320ALWB",
-        "SUBGRISM256",
-        "SUBGRISM128",
-        "SUBGRISM64",
-        "SUB32TATSGRISM",
-    ],
-    "niriss": [
-        "FULL",
-        "SUBSTRIP96",
-        "SUBSTRIP256",
-        "SUB80",
-        "SUB64",
-        "SUB128",
-        "SUB256",
-        "WFSS64R",
-        "WFSS128R",
-        "WFSS64C",
-        "WFSS128C",
-        "SUBAMPCAL",
-        "SUBTAAMI",
-        "SUBTASOSS",
-    ],
-    "nirspec": [],
-    "miri": [
-        "BRIGHTSKY",
-        "FULL",
-        "MASK1065",
-        "MASK1140",
-        "MASK1550",
-        "MASKLYOT",
-        "SLITLESSPRISM",
-        "SUB64",
-        "SUB128",
-        "SUB256",
-    ],
-    "fgs": [],
-}
+    "nircam": ['FULL'] + sorted(schema["properties"]["meta"]["properties"]["subarray"]["properties"]["name"]["anyOf"][2]['enum']),
+    "niriss": ['FULL'] + sorted(schema["properties"]["meta"]["properties"]["subarray"]["properties"]["name"]["anyOf"][4]['enum']),
+    "nirspec": ['FULL'] + sorted(schema["properties"]["meta"]["properties"]["subarray"]["properties"]["name"]["anyOf"][6]['enum']),
+    "miri": ['FULL'] + sorted(schema["properties"]["meta"]["properties"]["subarray"]["properties"]["name"]["anyOf"][1]['enum']),
+    "fgs": ['FULL'] + sorted(schema["properties"]["meta"]["properties"]["subarray"]["properties"]["name"]["anyOf"][0]['enum'])
+    }
 
 # Filename suffixes that need to include the association value in the suffix in
 # order to identify the preview image file. This should only be crf and crfints,


Test patch:
Repository: spacetelescope/jwql

Issue:
Incorrect NIRSpec filter
Using the query form of JWQL and clicking on NIRSpec, the list of filters contains the entry P750L. This is a filter for MIRI, not NIRSpec and should be removed from the NIRSpec listing. This filter is present in the MIRI filter list.




Discussion:

Resolving patch:
diff --git a/jwql/utils/constants.py b/jwql/utils/constants.py
--- a/jwql/utils/constants.py
+++ b/jwql/utils/constants.py
@@ -470,7 +470,6 @@
         "F170LP",
         "F290LP",
         "OPAQUE",
-        "P750L",
     ],
 }
 


Test patch: