internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: zulip/zulip

Issue:
Open Graph og:url tags over-canonicalize all URLs back to the home page
Split from #12187:

> By the way, it turns out all our OpenGraph tags are [busted anyway](https://developers.facebook.com/tools/debug/sharing/?q=https%3A%2F%2Fchat.zulip.org%2Fapi%2Fincoming-webhooks-walkthrough) because we always set `og:url` to point to the home page (which redirects to /login, whose `og:url` points back to the home page).


Discussion:
Hello @zulip/server-misc members, this issue was labeled with the "area: portico" label, so you may want to check it out!

<!-- areaLabelAddition -->
I think we either want to delete our `og:url` tags (if that DTRT) or specify the current page's address (I think relative links don't work?); I'm not sure we have a use case for pointing to a different canonical URL from the current one right now.
@punchagan this may be worth looking at while you're working on open graph stuff.
@zulipbot claim

Resolving patch:
diff --git a/zerver/context_processors.py b/zerver/context_processors.py
--- a/zerver/context_processors.py
+++ b/zerver/context_processors.py
@@ -133,6 +133,7 @@ def zulip_default_context(request: HttpRequest) -> Dict[str, Any]:
         'allow_search_engine_indexing': allow_search_engine_indexing,
     }
 
+    context['OPEN_GRAPH_URL'] = '%s%s' % (realm_uri, request.path)
     if realm is not None and realm.icon_source == realm.ICON_UPLOADED:
         context['OPEN_GRAPH_IMAGE'] = '%s%s' % (realm_uri, realm_icon)
 


Test patch:
diff --git a/zerver/tests/test_middleware.py b/zerver/tests/test_middleware.py
--- a/zerver/tests/test_middleware.py
+++ b/zerver/tests/test_middleware.py
@@ -184,3 +184,13 @@ def test_login_page_realm_icon(self) -> None:
         twitter_image = bs.select_one('meta[name="twitter:image"]').get('content')
         self.assertTrue(open_graph_image.endswith(realm_icon))
         self.assertTrue(twitter_image.endswith(realm_icon))
+
+    def test_no_realm_api_page_og_url(self) -> None:
+        response = self.client_get('/api/', subdomain='')
+        self.assertEqual(response.status_code, 200)
+
+        decoded = response.content.decode('utf-8')
+        bs = BeautifulSoup(decoded, features='lxml')
+        open_graph_url = bs.select_one('meta[property="og:url"]').get('content')
+
+        self.assertTrue(open_graph_url.endswith('/api/'))
Repository: zulip/zulip

Issue:
Scrollbar drag can result in unintended click actions
Split off from #11792:

> * on the settings pages, if you click on the scrollbar, drag it down, and then release your click when the mouse is outside the settings modal (e.g. below it or to the right), it closes the settings modal. I don't know if this is an existing thing or a regression, but I ran into it a bunch of times when testing even after knowing the behavior.

This was not a regression from perfect-scrollbar, but I fixed it in Grsmto/simplebar#312 and Grsmto/simplebar#317. Just waiting for the fixes to be included in a new upstream release.


Discussion:

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -11,4 +11,4 @@
 # Typically, adding a dependency only requires a minor version bump, and
 # removing a dependency requires a major version bump.
 
-PROVISION_VERSION = '32.0'
+PROVISION_VERSION = '32.1'


Test patch:
Repository: zulip/zulip

Issue:
Clean up `update-locked-requirements` and `requirements.in` files to remove `-e` hackery.
It looks like https://github.com/jazzband/pip-tools/pull/807 was included in the latest `pip-tools` release 12 days ago.  I think this may mean we can get rid of our semantically incorrect usage of `-e` in our requirements files, which in turn may mean we can remove most of the messy code in `tools/update-locked-requirements` related to hackily removing the `-e` lines. 
 See `compile_requirements` in that file for details.  

My guess is that this means if we upgrade pip-tools, we can delete 50% of the code in `update-locked-requirements` and clean up our `requirements.in` files to not use `-e`.  

@hackerkid this might be a good project for you.
Clean up `update-locked-requirements` and `requirements.in` files to remove `-e` hackery.
It looks like https://github.com/jazzband/pip-tools/pull/807 was included in the latest `pip-tools` release 12 days ago.  I think this may mean we can get rid of our semantically incorrect usage of `-e` in our requirements files, which in turn may mean we can remove most of the messy code in `tools/update-locked-requirements` related to hackily removing the `-e` lines. 
 See `compile_requirements` in that file for details.  

My guess is that this means if we upgrade pip-tools, we can delete 50% of the code in `update-locked-requirements` and clean up our `requirements.in` files to not use `-e`.  

@hackerkid this might be a good project for you.


Discussion:
Hello @zulip/server-tooling members, this issue was labeled with the "area: tooling" label, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot claim
Hello @zulip/server-tooling members, this issue was labeled with the "area: tooling" label, so you may want to check it out!

<!-- areaLabelAddition -->
@zulipbot claim

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -26,4 +26,4 @@
 #   historical commits sharing the same major version, in which case a
 #   minor version bump suffices.
 
-PROVISION_VERSION = '49.1'
+PROVISION_VERSION = '49.2'


Test patch:
Repository: zulip/zulip

Issue:
Upgrade pip from 19.1.1 and pip-tools from 3.8.0
Followup issue from #13067. pip-tools 3.9.0 or 4.0.0 fails to resolve dependencies from Git URLs (jazzband/pip-tools#851):

`pip._internal.exceptions.DistributionNotFound: No matching distribution found for zulip==0.6.1_git (from -r requirements/common.in (line 135))`

while pip 19.2 breaks pip-tools 3.8.0 (jazzband/pip-tools#853):

`TypeError: __init__() got an unexpected keyword argument 'find_links'`


Discussion:
Hello @zulip/server-dependencies members, this issue was labeled with the "area: dependencies" label, so you may want to check it out!

<!-- areaLabelAddition -->
I opened a possible fix as jazzband/pip-tools#879; merged for the next pip-tools release (presumably 4.1.0).

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -26,4 +26,4 @@
 #   historical commits sharing the same major version, in which case a
 #   minor version bump suffices.
 
-PROVISION_VERSION = '49.2'
+PROVISION_VERSION = '49.3'


Test patch:
Repository: zulip/zulip

Issue:
Don't generate useless local_database_password secret in production
The `local_database_password` secret is only used in the Zulip development environment, and it's a bug that `generate_secrets.py` generates it in production.  We should ensure it doesn't get generated there for new installations, since it's confusing.  

It might also be helpful to make `generate_secrets.py` remove or comment it on upgrade, though I suppose that carries some risk if the user has written their own code to access that setting.  

@andersk @mateuszmandera FYI


Discussion:
Hello @zulip/server-production members, this issue was labeled with the "area: production installer" label, so you may want to check it out!

<!-- areaLabelAddition -->

Resolving patch:
diff --git a/scripts/setup/generate_secrets.py b/scripts/setup/generate_secrets.py
--- a/scripts/setup/generate_secrets.py
+++ b/scripts/setup/generate_secrets.py
@@ -26,7 +26,6 @@
 AUTOGENERATED_SETTINGS = [
     'avatar_salt',
     'initial_password_salt',
-    'local_database_password',
     'rabbitmq_password',
     'shared_secret',
     'thumbor_key',
@@ -85,6 +84,9 @@ def add_secret(name, value):
         if need_secret(name):
             add_secret(name, generate_random_token(64))
 
+    if development and need_secret("local_database_password"):
+        add_secret("local_database_password", generate_random_token(64))
+
     if need_secret('secret_key'):
         add_secret('secret_key', generate_django_secretkey())
 
diff --git a/zproject/config.py b/zproject/config.py
--- a/zproject/config.py
+++ b/zproject/config.py
@@ -28,9 +28,7 @@ def get_secret(key: str, default_value: Optional[str]=None,
                development_only: bool=False) -> Optional[str]:
     if development_only and PRODUCTION:
         return default_value
-    if secrets_file.has_option('secrets', key):
-        return secrets_file.get('secrets', key)
-    return default_value
+    return secrets_file.get('secrets', key, fallback=default_value)
 
 @overload
 def get_config(section: str, key: str, default_value: str) -> str:
@@ -39,9 +37,7 @@ def get_config(section: str, key: str, default_value: str) -> str:
 def get_config(section: str, key: str, default_value: Optional[str]=None) -> Optional[str]:
     ...
 def get_config(section: str, key: str, default_value: Optional[str]=None) -> Optional[str]:
-    if config_file.has_option(section, key):
-        return config_file.get(section, key)
-    return default_value
+    return config_file.get(section, key, fallback=default_value)
 
 def get_from_file_if_exists(path: str) -> str:
     if os.path.exists(path):


Test patch:
Repository: zulip/zulip

Issue:
Creation of temporary files in requirements/ can cause provision to fail
An example to trigger this for me is was as follows:
* `cd requirements/`
* edit file using editor which creates temporary file in this location (eg vim, depending on configuration)
* `tools/provision`
* provision fails with an error like
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcd in position 17: invalid continuation byte
```

This appears to be due to the venv management script not being able to handle the unexpected file produced by eg. vim.

This is not a major issue, but is a bit of a strange issue to debug if you are not expecting it or are new, and potentially could be easy to fix.


Discussion:
Can you provide the full traceback and the filename?  We probably are globbing for `.txt` files in that directory as part of our logic to check if anything has changed in the configuration.

Resolving patch:
diff --git a/scripts/lib/clean_venv_cache.py b/scripts/lib/clean_venv_cache.py
--- a/scripts/lib/clean_venv_cache.py
+++ b/scripts/lib/clean_venv_cache.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 import argparse
+import glob
 import os
 import sys
 
@@ -40,8 +41,8 @@ def add_current_venv_cache(venv_name: str) -> None:
         # list its requirements subdirectory.
         if not os.path.exists(reqs_dir):
             continue
-        for filename in os.listdir(reqs_dir):
-            requirements_file = os.path.join(reqs_dir, filename)
+        requirements_files = glob.glob(os.path.join(reqs_dir, "*.txt"))
+        for requirements_file in requirements_files:
             deps = expand_reqs(requirements_file)
             hash_val = hash_deps(deps)
             caches_in_use.add(os.path.join(VENV_CACHE_DIR, hash_val))


Test patch:
Repository: zulip/zulip

Issue:
Zulip improperly uses `lstrip` to remove prefix in development integrations
Hello,
While [scanning a bunch of code](https://app.r2c.dev), I came across https://github.com/zulip/zulip/blob/master/zerver/views/development/integrations.py#L76 which uses `lstrip` to remove a prefix from HTTP headers.

`lstrip` is misleading -- `lstrip` removes a **set**, not a **sequence**, of characters from the left side of the string. (https://docs.python.org/3/library/stdtypes.html#str.lstrip)

You can see the behavior of `lstrip` below:

```python
>>> tests = (
... "HTTP_HEADER",
... "HTP_PADDING",
... "TPS_REPORT",
... "SAFE_STRING",
... "__PRIVATE_VAR",
... "PTHPHPTPPHPTPHPT___REMAIN"
... )
>>> for test in tests:
... print(test.lstrip("HTTP_"))
... 
EADER     #Yikes
ADDING
S_REPORT
SAFE_STRING
RIVATE_VAR
REMAIN
>>> 
```

Surprisingly, a quick Google search shows that there is no standard method of removing a prefix in Python. [This post](https://stackoverflow.com/a/16892491) on Stack Overflow offers the following one-liner as a fix:

```python
text[text.startswith(prefix) and len(prefix):]
```

Anyway, I hope this helps!  If you prefer, I can open a pull request to patch the issue, but I figured someone else would have the development and testing environments already set up and could fix it faster than I could.  :)
Thanks!


Discussion:

Resolving patch:
diff --git a/zerver/views/development/integrations.py b/zerver/views/development/integrations.py
--- a/zerver/views/development/integrations.py
+++ b/zerver/views/development/integrations.py
@@ -70,13 +70,13 @@ def get_fixtures(request: HttpResponse,
 
         headers_raw = get_fixture_http_headers(integration_name,
                                                "".join(fixture.split(".")[:-1]))
-        headers = {}
-        for header in headers_raw:
+
+        def fix_name(header: str) -> str:
             if header.startswith("HTTP_"):  # HTTP_ is a prefix intended for Django.
-                headers[header.lstrip("HTTP_")] = headers_raw[header]
-            else:
-                headers[header] = headers_raw[header]
+                return header[len("HTTP_"):]
+            return header
 
+        headers = {fix_name(k): v for k, v in headers_raw.items()}
         fixtures[fixture] = {"body": body, "headers": headers}
 
     return json_success({"fixtures": fixtures})
diff --git a/zerver/views/documentation.py b/zerver/views/documentation.py
--- a/zerver/views/documentation.py
+++ b/zerver/views/documentation.py
@@ -107,7 +107,7 @@ def get_context_data(self, **kwargs: Any) -> Dict[str, Any]:
             with open(article_path) as article_file:
                 first_line = article_file.readlines()[0]
             # Strip the header and then use the first line to get the article title
-            article_title = first_line.strip().lstrip("# ")
+            article_title = first_line.lstrip("#").strip()
             if context["not_index_page"]:
                 context["OPEN_GRAPH_TITLE"] = "%s (%s)" % (article_title, title_base)
             else:


Test patch:
Repository: zulip/zulip

Issue:
Remove the TODO "7 days" restriction for edit and move topics
Right now we have a restriction to move just the messages in the last week in method:
`update_messages_for_topic_edit` file `zerver/lib/topic.py`
```
       # We only change messages up to 7 days in the past, to avoid hammering our
        # DB by changing an unbounded amount of messages
        #
        # TODO: Look at removing this restriction and/or add a "change_last_week"
        # option; this behavior feels buggy.
```


Discussion:
Hello @zulip/server-message-view members, this issue was labeled with the "area: message-editing" label, so you may want to check it out!

<!-- areaLabelAddition -->
Tagging this as a release blocker to make sure we look at it before releasing with #13912.
@zulipbot claim

Resolving patch:
diff --git a/zerver/lib/topic.py b/zerver/lib/topic.py
--- a/zerver/lib/topic.py
+++ b/zerver/lib/topic.py
@@ -1,8 +1,5 @@
-import datetime
-
 from django.db import connection
 from django.db.models.query import QuerySet, Q
-from django.utils.timezone import now as timezone_now
 
 from sqlalchemy.sql import (
     column,
@@ -124,15 +121,7 @@ def update_messages_for_topic_edit(message: Message,
                                    new_stream: Optional[Stream]) -> List[Message]:
     propagate_query = Q(recipient = message.recipient, subject = orig_topic_name)
     if propagate_mode == 'change_all':
-        # We only change messages up to 7 days in the past, to avoid hammering our
-        # DB by changing an unbounded amount of messages
-        #
-        # TODO: Look at removing this restriction and/or add a "change_last_week"
-        # option; this behavior feels buggy.
-        before_bound = timezone_now() - datetime.timedelta(days=7)
-
-        propagate_query = (propagate_query & ~Q(id = message.id) &
-                           Q(date_sent__range=(before_bound, timezone_now())))
+        propagate_query = propagate_query & ~Q(id = message.id)
     if propagate_mode == 'change_later':
         propagate_query = propagate_query & Q(id__gt = message.id)
 


Test patch:
Repository: zulip/zulip

Issue:
Remove stream from DefaultStreamGroup if the stream is deactivated
When we deactivate a stream which is part of an already existing DefaultStreamGroup, stream is not removed from the default stream group.

This is a bug and the stream should be removed from the default stream group on deactivation of the stream.


Discussion:
@zulipbot claim

Resolving patch:
diff --git a/zerver/lib/actions.py b/zerver/lib/actions.py
--- a/zerver/lib/actions.py
+++ b/zerver/lib/actions.py
@@ -851,6 +851,10 @@ def do_deactivate_stream(stream: Stream, log: bool=True) -> None:
     if DefaultStream.objects.filter(realm_id=stream.realm_id, stream_id=stream.id).exists():
         do_remove_default_stream(stream)
 
+    default_stream_groups_for_stream = DefaultStreamGroup.objects.filter(streams__id=stream.id)
+    for group in default_stream_groups_for_stream:
+        do_remove_streams_from_default_stream_group(stream.realm, group, [stream])
+
     # Remove the old stream information from remote cache.
     old_cache_key = get_stream_cache_key(old_name, stream.realm_id)
     cache_delete(old_cache_key)


Test patch:
diff --git a/zerver/tests/test_subs.py b/zerver/tests/test_subs.py
--- a/zerver/tests/test_subs.py
+++ b/zerver/tests/test_subs.py
@@ -456,6 +456,31 @@ def test_deactivate_stream_removes_default_stream(self) -> None:
         do_deactivate_stream(stream)
         self.assertEqual(0, DefaultStream.objects.filter(stream_id=stream.id).count())
 
+    def test_deactivate_stream_removes_stream_from_default_stream_groups(self) -> None:
+        realm = get_realm('zulip')
+        streams_to_keep = []
+        for stream_name in ["stream1", "stream2"]:
+            stream = ensure_stream(realm, stream_name)
+            streams_to_keep.append(stream)
+
+        streams_to_remove = []
+        stream = ensure_stream(realm, "stream3")
+        streams_to_remove.append(stream)
+
+        all_streams = streams_to_keep + streams_to_remove
+
+        def get_streams(group: DefaultStreamGroup) -> List[Stream]:
+            return list(group.streams.all().order_by('name'))
+
+        group_name = "group1"
+        description = "This is group1"
+        do_create_default_stream_group(realm, group_name, description, all_streams)
+        default_stream_groups = get_default_stream_groups(realm)
+        self.assertEqual(get_streams(default_stream_groups[0]), all_streams)
+
+        do_deactivate_stream(streams_to_remove[0])
+        self.assertEqual(get_streams(default_stream_groups[0]), streams_to_keep)
+
     def test_vacate_private_stream_removes_default_stream(self) -> None:
         stream = self.make_stream('new_stream', invite_only=True)
         self.subscribe(self.example_user("hamlet"), stream.name)
Repository: zulip/zulip

Issue:
JPEG images with suffix jpe are not shown as image
When copying an image on iOS (photo library) and pasting this image on Zulip running on Safari on a Mac (Universal clipboard feature), Zulip will detect that it is a graphic, upload the file and generate a reference like

    [pasted image](https://server.example.com/user_uploads/hash/pasted_image.jpe)

The file has an .jpe ending, which is probably not the most standard, but given that it says, ‘pasted image’, I assume that some part of Zulip realised that this is an image.

However, this markdown is then only converted into a link. The image file itself is never embedded. To fix the problem, I have to download the file again on the Mac, change the suffix and upload it again. It would be nicer, if there were more ways to force the Markdown to do as I tell it.


Discussion:
I bet we just need to add that extension in our lists of image extensions.

Thanks for the report!
Hello @zulip/server-misc members, this issue was labeled with the "area: uploads" label, so you may want to check it out!

<!-- areaLabelAddition -->
Places to change likely include:
* `zerver/lib/bugdown/__init__.py`
* `puppet/zulip/files/nginx/zulip-include-frontend/uploads.types`
* `zerver/lib/upload.py`
* Various other things found via `git grep --files-with-match jpeg  | xargs grep gif`

Resolving patch:
diff --git a/zerver/lib/bugdown/__init__.py b/zerver/lib/bugdown/__init__.py
--- a/zerver/lib/bugdown/__init__.py
+++ b/zerver/lib/bugdown/__init__.py
@@ -667,7 +667,7 @@ def is_image(self, url: str) -> bool:
             return False
 
         # List from https://support.google.com/chromeos/bin/answer.py?hl=en&answer=183093
-        for ext in [".bmp", ".gif", ".jpg", "jpeg", ".png", ".webp"]:
+        for ext in [".bmp", ".gif", ".jpe", "jpeg", ".jpg", ".png", ".webp"]:
             if parsed_url.path.lower().endswith(ext):
                 return True
         return False


Test patch:
Repository: zulip/zulip

Issue:
Redirect portico pages to the root domain
Redirect almost all pages affected by #14898 to `ROOT_DOMAIN_URI` with permanent redirects:

* /api
* /apps
* /atlassian
* /features
* /for/companies
* /for/mystery-hunt
* /for-open-source
* /for/working-groups-and-communities
* /hello
* /help
* /history
* /integrations
* /privacy
* /security
* /team
* /terms
* /why-zulip

Not /plans, /plans is up to something weird.

**Testing Plan:** Dev server (where we actually redirect to `localhost:9991` rather than `ROOT_DOMAIN_URI` under the same logic we used for `STATIC_ROOT`).


Discussion:

Resolving patch:
diff --git a/zerver/data_import/slack.py b/zerver/data_import/slack.py
--- a/zerver/data_import/slack.py
+++ b/zerver/data_import/slack.py
@@ -1033,6 +1033,8 @@ def do_convert_data(slack_zip_file: str, output_dir: str, token: str, threads: i
     realm_id = 0
     domain_name = settings.EXTERNAL_HOST
 
+    log_token_warning(token)
+
     slack_data_dir = slack_zip_file.replace('.zip', '')
     if not os.path.exists(slack_data_dir):
         os.makedirs(slack_data_dir)
@@ -1100,15 +1102,16 @@ def get_data_file(path: str) -> Any:
         data = ujson.load(fp)
         return data
 
+def log_token_warning(token: str) -> None:
+    if not token.startswith("xoxp-"):
+        logging.info('Not a Slack legacy token.\n'
+                     '  This token might not have all the needed scopes. We need the following scopes:\n'
+                     '  - emoji:read\n  - users:read\n  - users:read.email\n  - team:read')
+
+
 def get_slack_api_data(slack_api_url: str, get_param: str, **kwargs: Any) -> Any:
     if not kwargs.get("token"):
         raise AssertionError("Slack token missing in kwargs")
-    token = kwargs["token"]
-    if not token.startswith("xoxp-"):
-        raise Exception('Invalid Slack legacy token.\n'
-                        '  You must pass a Slack "legacy token" starting with "xoxp-".\n'
-                        '  Create one at https://api.slack.com/custom-integrations/legacy-tokens')
-
     data = requests.get("{}?{}".format(slack_api_url, urlencode(kwargs)))
 
     if data.status_code == requests.codes.ok:


Test patch:
diff --git a/zerver/tests/test_slack_importer.py b/zerver/tests/test_slack_importer.py
--- a/zerver/tests/test_slack_importer.py
+++ b/zerver/tests/test_slack_importer.py
@@ -96,11 +96,6 @@ def test_get_slack_api_data(self, mock_get: mock.Mock) -> None:
             get_slack_api_data(slack_user_list_url, "members", token=token)
         self.assertEqual(invalid.exception.args, ('Error accessing Slack API: invalid_auth',),)
 
-        token = 'xoxe-invalid-token'
-        with self.assertRaises(Exception) as invalid:
-            get_slack_api_data(slack_user_list_url, "members", token=token)
-        self.assertTrue(invalid.exception.args[0].startswith("Invalid Slack legacy token.\n"))
-
         with self.assertRaises(Exception) as invalid:
             get_slack_api_data(slack_user_list_url, "members")
         self.assertEqual(invalid.exception.args, ('Slack token missing in kwargs',),)