internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: hydroshare/hydroshare

Issue:
recommended metadata message shouldn't request readme for alt res types
**Description of the bug**
recommended metadata message on resource landing page: For collection and app connector we are recommending the user to upload a readme file. This recommendation only applies to composite resource as the user is not allowed to upload files to collection or app connector.
![image](https://user-images.githubusercontent.com/17934193/207676099-857c712d-fabe-4602-98d0-e31626f8968b.png)

Steps to reproduce the bug:
1. Create a web-app connector
2. See recommended metadata message that requests Readme.

**Expected behavior**
Uploading a readme isn't possible for these resources. It shouldn't be requested



Discussion:

Resolving patch:
diff --git a/hs_core/models.py b/hs_core/models.py
--- a/hs_core/models.py
+++ b/hs_core/models.py
@@ -4323,7 +4323,7 @@ def get_recommended_missing_elements(self):
         missing_recommended_elements = []
         if not self.funding_agencies.count():
             missing_recommended_elements.append('Funding Agency')
-        if not self.resource.readme_file:
+        if not self.resource.readme_file and self.resource.resource_type == "CompositeResource":
             missing_recommended_elements.append('Readme file containing variables, '
                                                 'abbreviations/acronyms, and non-standard file formats')
         if not self.coverages.count():


Test patch:
Repository: hydroshare/hydroshare

Issue:
prevent timezone warnings during session logging
**Describe the feature you'd like and what it will do**
Most actions within HS result in at least 4 RuntimeWarnings to be logged
Example creating a resource:
![image](https://user-images.githubusercontent.com/17934193/209165319-7e97eb58-2e23-4ee1-9095-0c5beeef0a8e.png)
This makes it difficult to sift through the logs and find what we need.
Get rid of these unnecessary warnings

**Why is this feature important?**
Make debugging and audit trail easier



Discussion:

Resolving patch:
diff --git a/hs_tracking/models.py b/hs_tracking/models.py
--- a/hs_tracking/models.py
+++ b/hs_tracking/models.py
@@ -1,4 +1,5 @@
 from datetime import datetime, timedelta
+from django.utils import timezone
 
 from django.db import models
 from django.db.models import F
@@ -30,6 +31,7 @@ def for_request(self, request, user=None):
         if signed_id:
             tracking_id = signing.loads(signed_id)
             cut_off = datetime.now() - timedelta(seconds=SESSION_TIMEOUT)
+            cut_off = timezone.make_aware(cut_off)
             session = None
 
             try:


Test patch:
Repository: hydroshare/hydroshare

Issue:
A required state field for user profiles may not make sense for some countries
If you go to the profile page, you'll note that state is a required field.  It may benefit usability to remove the requirement.

https://www.quora.com/Is-there-any-country-without-states

<img width="621" alt="006-02" src="https://user-images.githubusercontent.com/43424907/118430129-5b23e100-b6a1-11eb-9b09-4288646ff6c0.png">

<img width="339" alt="006" src="https://user-images.githubusercontent.com/43424907/118429986-041e0c00-b6a1-11eb-9f06-85904cb7b96e.png">



Discussion:

Resolving patch:
diff --git a/theme/forms.py b/theme/forms.py
--- a/theme/forms.py
+++ b/theme/forms.py
@@ -233,7 +233,7 @@ class Meta:
     organization = forms.CharField(required=True)
     user_type = forms.CharField(required=True)
     country = forms.CharField(label='Country', required=True)
-    state = forms.CharField(label='State/Province', required=True)
+    state = forms.CharField(label='State/Region', required=True)
 
     def __init__(self, request, *args, **kwargs):
         self.request = request
@@ -280,8 +280,6 @@ def clean_country(self):
 
     def clean_state(self):
         data = self.cleaned_data['state']
-        if len(data.strip()) == 0:
-            raise forms.ValidationError("State is a required field.")
         return data
 
     def save(self, *args, **kwargs):
@@ -344,8 +342,6 @@ def clean_country(self):
 
     def clean_state(self):
         data = self.cleaned_data['state']
-        if data is None or len(data.strip()) == 0:
-            raise forms.ValidationError("State is a required field.")
         return data
 
     def clean_identifiers(self):


Test patch:
Repository: hydroshare/hydroshare

Issue:
phone numbers failing validation show ambiguous error messaging
**Description of the bug**
Entering a pone number exceeding 16char results in a sensible message:
"Ensure this value has at most 16 characters (it has 20)."
but that message is displayed _after submission_ so the user has no way of knowing what the message refers to.

Steps to reproduce the bug:
1. Update profile with a phone number > 16char
2. Save changes
3. See error message

https://user-images.githubusercontent.com/17934193/209988396-e2b55604-4f8f-4998-96cb-02f4ecfaea67.mov



**Expected behavior**
It should be clear what field the validation is failing on

**Additional information**
HS v2.0



Discussion:
related to #4235

Resolving patch:
diff --git a/theme/migrations/0021_auto_20221229_2200.py b/theme/migrations/0021_auto_20221229_2200.py
new file mode 100644
--- /dev/null
+++ b/theme/migrations/0021_auto_20221229_2200.py
@@ -0,0 +1,23 @@
+# Generated by Django 3.2.15 on 2022-12-29 22:00
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('theme', '0020_auto_20220922_1400'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='userprofile',
+            name='phone_1',
+            field=models.CharField(blank=True, max_length=32, null=True),
+        ),
+        migrations.AlterField(
+            model_name='userprofile',
+            name='phone_2',
+            field=models.CharField(blank=True, max_length=32, null=True),
+        ),
+    ]
diff --git a/theme/models.py b/theme/models.py
--- a/theme/models.py
+++ b/theme/models.py
@@ -2,7 +2,6 @@
 import os
 import logging
 
-from django.core.validators import RegexValidator
 from django.utils import timezone
 from django.dispatch import receiver
 from django.contrib.auth.models import User
@@ -320,15 +319,8 @@ class UserProfile(models.Model):
         blank=True,
         help_text="The name of the organization you work for.",
     )
-    phone_regex = RegexValidator(
-        regex=r"^\d{8,15}$",
-        message="Our validation for phone numbers has recently changed. Please ensure that "
-        "your phone number is entered in the following format: '999999999'. Up to 16 "
-        "digits are allowed.  If you made other changes to your profile - please make "
-        "sure they saved successfully.",
-    )
     phone_1 = models.CharField(
-        validators=[phone_regex], max_length=16, null=True, blank=True
+        max_length=32, null=True, blank=True
     )
     phone_1_type = models.CharField(
         max_length=1024,
@@ -341,7 +333,7 @@ class UserProfile(models.Model):
         ),
     )
     phone_2 = models.CharField(
-        validators=[phone_regex], max_length=16, null=True, blank=True
+        max_length=32, null=True, blank=True
     )
     phone_2_type = models.CharField(
         max_length=1024,


Test patch:
Repository: hydroshare/hydroshare

Issue:
update our base image dependencies
**Describe the feature you'd like and what it will do**
As part of the [effort to update our base image](https://github.com/hydroshare/hs_docker_base/pull/71), I updated many of [our dependencies](https://github.com/hydroshare/hs_docker_base/blob/2.2.0/requirements.txt) however there are still quite a few that need updating.

We should focus on those that have known high-severity vulnerabilities first.

**Why is this feature important?**
Let's keep things up to date with an eye for security.

**Additional context**
HS v2.1



Discussion:

Resolving patch:
diff --git a/django_irods/tasks.py b/django_irods/tasks.py
--- a/django_irods/tasks.py
+++ b/django_irods/tasks.py
@@ -41,8 +41,8 @@
 * ixmsg    - send/receive iRODS xMessage System messages.
 """
 
-from celery.task import Task
-from celery.task import subtask
+from celery import Task
+from celery import subtask
 from .icommands import Session, GLOBAL_SESSION, IRodsEnv
 
 from . import models as m
diff --git a/hs_core/task_utils.py b/hs_core/task_utils.py
--- a/hs_core/task_utils.py
+++ b/hs_core/task_utils.py
@@ -1,4 +1,4 @@
-from celery.task.control import inspect
+from hydroshare.hydrocelery import app as celery_app
 from celery.result import AsyncResult
 from django.db import transaction
 
@@ -8,7 +8,7 @@
 
 logger = logging.getLogger('django')
 
-celery_inspector = inspect()
+celery_inspector = celery_app.control.inspect()
 
 
 def _retrieve_task_id(job_name, res_id, job_dict):
diff --git a/hydroshare/hydrocelery.py b/hydroshare/hydrocelery.py
--- a/hydroshare/hydrocelery.py
+++ b/hydroshare/hydrocelery.py
@@ -10,7 +10,8 @@
 os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hydroshare.settings')
 os.environ.setdefault('PYTHONPATH', '/hydroshare/hydroshare')
 
-app = Celery('hydroshare', backend='amqp://')
+# ampq backend has been deprecated in celery v5
+app = Celery('hydroshare', backend='rpc://', broker=settings.BROKER_URL)
 app.config_from_object('django.conf:settings')
 app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
 


Test patch:
Repository: hydroshare/hydroshare

Issue:
Sharing Status can be modified during pending metadata review
**Description of the bug**
https://user-images.githubusercontent.com/17934193/214396124-257d4f8e-1a2e-4bfb-99f8-0957595f6905.mov
![image](https://user-images.githubusercontent.com/17934193/211364805-57d71e22-51e0-4a26-b4a9-d98aa45d49a3.png)
After requesting metadata review, the sharing status can be changed from "Public" to "Discoverable/Private"

Steps to reproduce the bug:
Example: https://www.hydroshare.org/resource/c1fc92e773564ad6b4687f803e32736a/
1. On a public resource, request metadata review
2. See that you are able to change the sharing status

**Expected behavior**
Sharing status should remain "Public" during metadata review

**Additional information**
HS v2.1



Discussion:
related to https://github.com/hydroshare/hydroshare/issues/4726

Resolving patch:
diff --git a/hs_core/views/__init__.py b/hs_core/views/__init__.py
--- a/hs_core/views/__init__.py
+++ b/hs_core/views/__init__.py
@@ -1190,6 +1190,9 @@ def publish(request, shortkey, *args, **kwargs):
     if not request.user.is_superuser:
         raise ValidationError("Resource can only be published by an admin user")
     try:
+        res = get_resource_by_shortkey(shortkey)
+        res.raccess.review_pending = False
+        res.raccess.save()
         hydroshare.publish_resource(request.user, shortkey)
     except ValidationError as exp:
         request.session["validation_error"] = str(exp)
@@ -1221,7 +1224,9 @@ def submit_for_review(request, shortkey, *args, **kwargs):
                 Your resource is under review for appropriate minimum metadata and to ensure that it adheres to
                 community guidelines.
                 The review process will likely be complete within 1 business day, but not exceed 2 business days.
-                You will receive a notification via email once the review process has concluded."""
+                You will receive a notification via email once the review process has concluded.
+                If you decide that you no longer wish to have this resource reviewed for publication,
+                please contact help@cuahsi.org"""
         messages.success(request, message)
     return HttpResponseRedirect(request.META["HTTP_REFERER"])
 
@@ -2113,7 +2118,7 @@ def metadata_review(request, shortkey, action, uidb36=None, token=None, **kwargs
                 request,
                 "Publication request was rejected. Please send an email to the resource owner indicating why.",
             )
-            res.metadata.dates.all().filter(type="review_started").delete()
+        res.metadata.dates.all().filter(type="review_started").delete()
     return HttpResponseRedirect(f"/resource/{ res.short_id }/")
 
 


Test patch:
Repository: hydroshare/hydroshare

Issue:
HS dev environments should expose schema.org but prevent crawling
Important for testing esp with DSP/Discovery

I also see that we don't have ROBOTS_SITEMAP_URLS set on prod or beta, resulting in:
```
User-agent: *
Disallow: /hsapi
Disallow: /o/
Disallow: /accounts/login/
Disallow: /django_irods/download/
Disallow: /sign-up/
Disallow: /tracking/applaunch/
Crawl-delay: 5

Host: www.hydroshare.org
Sitemap: http://localhost:8000/sitemap/
```
https://www.hydroshare.org/robots.txt
(note the bad sitemap)


**Additional context**
HSv2.3



Discussion:
related to
https://github.com/hydroshare/hydroshare/issues/3505
https://github.com/hydroshare/hydroshare/issues/3478
https://github.com/hydroshare/hydroshare/issues/1797
https://github.com/hydroshare/hydroshare/issues/1708

see:
https://developers.google.com/search/docs/crawling-indexing/block-indexing

Resolving patch:
diff --git a/hs_core/management/commands/prevent_web_crawling.py b/hs_core/management/commands/prevent_web_crawling.py
new file mode 100644
--- /dev/null
+++ b/hs_core/management/commands/prevent_web_crawling.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+
+"""
+Prevent web-crawlers from indexing the site
+"""
+
+from django.core.management.base import BaseCommand
+from django.contrib.sites.models import Site
+from robots.models import Url, Rule
+
+
+class Command(BaseCommand):
+    help = "Prevent web-crawlers from indexing the site. This command is intended to be run after dev deployments."
+
+    def add_arguments(self, parser):
+
+        # a list of urls to disable
+        parser.add_argument('disallowed_urls', nargs='*', type=str)
+
+    def handle(self, *args, **options):
+        # django-robots model definitions:
+        # https://github.com/jazzband/django-robots/blob/5.0/src/robots/models.py
+        r = Rule()
+        r.robot = "*"
+        r.crawl_delay = 5
+        r.save()  # extra save here is necessary to do r.disallowed later
+        r.sites.set(Site.objects.all())
+
+        if len(options['disallowed_urls']) > 0:  # an array of urls to disable.
+            for url in options['disallowed_urls']:
+                try:
+                    u = Url()
+                    u.pattern = url
+                    u.save()
+                    r.disallowed.add(u)
+                except Exception:
+                    print(f"Disallow url: {url} failed.")
+                    raise
+        else:
+            u = Url()
+            u.pattern = "/"
+            u.save()
+            r.disallowed.add(u)
+        r.save()


Test patch:
Repository: hydroshare/hydroshare

Issue:
Resources with Organization as Author should show this author in My Resources
**Description of the bug**
The My Resources Listing for a resource owned by an organization, lists Author as None.  It should list the Organization Author.
![image](https://user-images.githubusercontent.com/3456833/217813270-e98f233f-f251-49d0-ae77-de17afbed612.png)


Steps to reproduce the bug:
1. Go to a resources such as https://www.hydroshare.org/resource/5057577e8573433d8045b59db91b2550/
2. Add to My Resources

![image](https://user-images.githubusercontent.com/3456833/217813530-3c18b0d6-f917-47e9-b0ce-f09b967159ca.png)

4. Go to My Resources
5. See error (My resources listing omits author)

**Expected behavior**
The Organization Author should be listed under author.




Discussion:
adding some additional info
Related to PR https://github.com/hydroshare/hydroshare/pull/3011 and issues:
https://github.com/hydroshare/hydroshare/issues/2941
https://github.com/hydroshare/hydroshare/issues/2931

Here is the template: https://github.com/hydroshare/hydroshare/blob/2.3/theme/templates/includes/my-resources-trows.html#L60

We modified the template tag most recently in [1.58](https://github.com/hydroshare/hydroshare/releases/tag/1.58), diff:
https://github.com/hydroshare/hydroshare/pull/4717/files#diff-37be798db1c0fc3349b2c258a83c19825d4a3bd707870dc4f1cc4ef3e8a07bf2R97
as part of work on https://github.com/hydroshare/hydroshare/issues/2368


Resolving patch:
diff --git a/hs_core/templatetags/hydroshare_tags.py b/hs_core/templatetags/hydroshare_tags.py
--- a/hs_core/templatetags/hydroshare_tags.py
+++ b/hs_core/templatetags/hydroshare_tags.py
@@ -116,21 +116,17 @@ def resource_first_author(content):
             first_creator = creator
             break
 
-    if first_creator:
-        if first_creator.name and first_creator.relative_uri and first_creator.is_active_user:
+    if first_creator.name:
+        if first_creator.relative_uri and first_creator.is_active_user:
             return format_html('<a href="{desc}">{name}</a>',
                                desc=first_creator.relative_uri,
                                name=first_creator.name)
-        elif first_creator.name:
-            return format_html('<span>{name}</span>', name=first_creator.name)
-    else:
-        first_creator = content.metadata.creators.filter(order=1).first()
-        if first_creator.name:
+        else:
             return format_html('<span>{name}</span>', name=first_creator.name)
-        if first_creator.organization:
-            return format_html('<span>{name}</span>', name=first_creator.organization)
+    elif first_creator.organization:
+        return format_html('<span>{name}</span>', name=first_creator.organization)
 
-        return ''
+    return ''
 
 
 @register.filter


Test patch:
Repository: hydroshare/hydroshare

Issue:
Geoconnex schema:name missing from some collection contexts
The issue is that the Geoconnex API returns a schema context that doesn't always match the actual features.
HS should gracefully handle when there is a mismatch between the schema and the features.


Discussion:

Resolving patch:
diff --git a/hs_tools_resource/middleware.py b/hs_tools_resource/middleware.py
--- a/hs_tools_resource/middleware.py
+++ b/hs_tools_resource/middleware.py
@@ -1,7 +1,8 @@
-from django.http import JsonResponse, HttpResponse
+from django.http import JsonResponse
 from django.core.exceptions import RequestDataTooBig
 from django.conf import settings
 from hs_tools_resource.utils import convert_size
+from django.urls import resolve
 
 
 class CheckRequest():
@@ -17,25 +18,8 @@ def __call__(self, request):
         try:
             _ = request.body
         except RequestDataTooBig:
-            limit = convert_size(settings.DATA_UPLOAD_MAX_MEMORY_SIZE)
-            return JsonResponse({"message": f"The upload was too big. Limit: {limit}"})
-
-        response = self.get_response(request)
-        return response
-
-    def process_request(self, request):
-        try:
-            request.body
-        except RequestDataTooBig as e:
-            response = self.process_exception(request, e)
-            if response is not None:
-                return response
-            else:
-                pass
-
-        return None
-
-    def process_exception(self, request, exception):
-        if isinstance(exception, RequestDataTooBig):
-            return HttpResponse(status=413)
-        return None
+            view_name = resolve(request.path).view_name
+            if view_name == "update_metadata_element":
+                limit = convert_size(settings.DATA_UPLOAD_MAX_MEMORY_SIZE)
+                return JsonResponse({"message": f"The upload was too big. Limit: {limit}"})
+        return self.get_response(request)
diff --git a/theme/views.py b/theme/views.py
--- a/theme/views.py
+++ b/theme/views.py
@@ -116,6 +116,7 @@ def get_context_data(self, **kwargs):
             "resources": resources,
             "quota_message": get_quota_message(u),
             "group_membership_requests": group_membership_requests,
+            "data_upload_max": settings.DATA_UPLOAD_MAX_MEMORY_SIZE
         }
 
 


Test patch:
Repository: hydroshare/hydroshare

Issue:
Haystack rest endpoint response serializer does not include short_id
**Description of the bug**
The Haystack REST endpoint for complex solr searches does not include the short_id into the response serializer. This is a critical piece of information for users of this endpoint.   

Steps to reproduce the bug:
https://github.com/hydroshare/hydroshare/blob/d3bd1737a0179eac74cd68926b3b79b80894410e/hs_rest_api/discovery.py#L12

**Expected behavior**
I expect resource ids to be included with search results so I can retrieve resources.



Discussion:

Resolving patch:
diff --git a/hs_rest_api/discovery.py b/hs_rest_api/discovery.py
--- a/hs_rest_api/discovery.py
+++ b/hs_rest_api/discovery.py
@@ -13,6 +13,7 @@ class DiscoveryResourceSerializer(HaystackSerializer):
     class Meta:
         index_classes = [BaseResourceIndex]
         fields = [
+            "short_id",
             "title",
             "author",
             "contributor",


Test patch:
Repository: hydroshare/hydroshare

Issue:
Resource file list endpoint performance improvements
Currently the endpoint (**/hsapi/resource/{id}/files**) is extremely slow. For listing 100 files it takes around **30 sec**. For each file we obtain the **checksum** and file **modified time** from **irods** as part of the file metadata response. Locally, when I turned off getting these values from irods, I was able to list 100 files in less than **1 sec**. In order to get this level of performance improvement, we need to store these values (file checksum and modified time) in Django **database**. 


Discussion:

Resolving patch:
diff --git a/django_irods/storage.py b/django_irods/storage.py
--- a/django_irods/storage.py
+++ b/django_irods/storage.py
@@ -436,7 +436,13 @@ def size(self, name):
             raise ValidationError(
                 "{} cannot be found in iRODS to retrieve " "file size".format(name)
             )
-        return int(float(stdout))
+        # remove potential '\n' from stdout
+        size_string = stdout.strip("0\n").replace("\n", "")
+        try:
+            ret = int(float(size_string))
+            return ret
+        except ValueError:
+            return 0
 
     def checksum(self, full_name, force_compute=True):
         """


Test patch: