internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: zulip/zulip

Issue:
markdown: Document built-in preprocessor priorities.
As a follow-up to #19783, it would be good to document the priorities assigned to the built-in preprocessors that the Python-Markdown library has. A couple of notes:
- This involves a bit of grunt work, the quickest way to do this is to loop over and print `md_engine.preprocessors._priorities` in `zerver/lib/templates.py`.
- Note that in `templates.py`, there are different cases where different sets of preprocessors are added, so one has to do the additional work to figure out which preprocessors are running in which of those cases and then document all the priorities that are for built-in preprocessors.
- The file to put these priorities in is: `zerver/lib/markdown/preprocessor_priorities..py`.

Thanks!


Discussion:
Hello @zulip/server-markdown members, this issue was labeled with the "area: markdown" label, so you may want to check it out!

<!-- areaLabelAddition -->

@zulipbot claim
Welcome to Zulip, @kevjn! We just sent you an invite to collaborate on this repository at https://github.com/zulip/zulip/invitations. Please accept this invite in order to claim this issue and begin a fun, rewarding experience contributing to Zulip!

Here's some tips to get you off to a good start:

- Join me on the [Zulip developers' server](https://chat.zulip.org), to get help, chat about this issue, and meet the other developers.
- [Unwatch this repository](https://help.github.com/articles/unwatching-repositories/), so that you don't get 100 emails a day.

As you work on this issue, you'll also want to refer to the [Zulip code contribution guide](https://zulip.readthedocs.io/en/latest/contributing/index.html), as well as the rest of the developer documentation on that site.

See you on the other side (that is, the pull request side)!


Resolving patch:
diff --git a/zerver/lib/markdown/preprocessor_priorities.py b/zerver/lib/markdown/preprocessor_priorities.py
--- a/zerver/lib/markdown/preprocessor_priorities.py
+++ b/zerver/lib/markdown/preprocessor_priorities.py
@@ -1,6 +1,7 @@
 # Note that in the Markdown preprocessor registry, the highest
 # numeric value is considered the highest priority, so the dict
 # below is ordered from highest-to-lowest priority.
+# Priorities for the built-in preprocessors are commented out.
 PREPROCESSOR_PRIORITES = {
     "generate_parameter_description": 535,
     "generate_response_description": 531,
@@ -10,9 +11,12 @@
     "generate_return_values": 510,
     "generate_api_arguments": 505,
     "include": 500,
+    # "include_wrapper": 500,
     "help_relative_links": 475,
     "setting": 450,
+    # "normalize_whitespace": 30,
     "fenced_code_block": 25,
+    # "html_block": 20,
     "tabbed_sections": -500,
     "nested_code_blocks": -500,
     "emoticon_translations": -505,


Test patch:
Repository: zulip/zulip

Issue:
Resolve topic message should use server's native language
When a user has set their language to a language that doesn't match the language in which people communicate, notification bot will post a message in the user's language instead of the language used for communication. In the case where I noticed it, notification bot posted the french message "`@<censored>` a marqué ce sujet comme résolu." on an english server. I think the message posted should either be in a language set globally for the server or be translated for every user.


Discussion:
Agreed; I feel like this came up in the last week for another reason as well.  This should be addressed by using `with override_language(realm.default_language)` like we do other messages by Notification Bot.  `maybe_send_resolve_topic_notifications` in `zerver/lib/actions.py` is the location of the code in question.
Hello @zulip/server-message-view members, this issue was labeled with the "area: message-editing" label, so you may want to check it out!

<!-- areaLabelAddition -->


Resolving patch:
diff --git a/zerver/lib/actions.py b/zerver/lib/actions.py
--- a/zerver/lib/actions.py
+++ b/zerver/lib/actions.py
@@ -5885,15 +5885,14 @@ def maybe_send_resolve_topic_notifications(
     if old_topic.lstrip(RESOLVED_TOPIC_PREFIX) != new_topic.lstrip(RESOLVED_TOPIC_PREFIX):
         return
 
-    if new_topic.startswith(RESOLVED_TOPIC_PREFIX) and not old_topic.startswith(
+    topic_resolved: bool = new_topic.startswith(RESOLVED_TOPIC_PREFIX) and not old_topic.startswith(
         RESOLVED_TOPIC_PREFIX
-    ):
-        notification_string = _("{user} has marked this topic as resolved.")
-    elif old_topic.startswith(RESOLVED_TOPIC_PREFIX) and not new_topic.startswith(
+    )
+    topic_unresolved: bool = old_topic.startswith(
         RESOLVED_TOPIC_PREFIX
-    ):
-        notification_string = _("{user} has marked this topic as unresolved.")
-    else:
+    ) and not new_topic.startswith(RESOLVED_TOPIC_PREFIX)
+
+    if not topic_resolved and not topic_unresolved:
         # If there's some other weird topic that does not toggle the
         # state of "topic starts with RESOLVED_TOPIC_PREFIX", we do
         # nothing. Any other logic could result in cases where we send
@@ -5912,6 +5911,11 @@ def maybe_send_resolve_topic_notifications(
     sender = get_system_bot(settings.NOTIFICATION_BOT, user_profile.realm_id)
     user_mention = f"@_**{user_profile.full_name}|{user_profile.id}**"
     with override_language(stream.realm.default_language):
+        if topic_resolved:
+            notification_string = _("{user} has marked this topic as resolved.")
+        elif topic_unresolved:
+            notification_string = _("{user} has marked this topic as unresolved.")
+
         internal_send_stream_message(
             sender,
             stream,


Test patch:
Repository: zulip/zulip

Issue:
Problem with unzip and get slack user/team
Hi
We have a problem with unzip file
If folder name has symbols e.g.   ö or сyrillic, we have wrong folder name, like on screen
![image](https://user-images.githubusercontent.com/43493241/136048531-2ac29f51-b37d-4285-892c-fe9836d9ee0a.png)
I found implementation
![image](https://user-images.githubusercontent.com/43493241/136049182-7c3c79a1-690e-40ab-93d4-a9e6f969b846.png)
I commented subprocess and add zipfile 
Maybe its a better and you can add to master
I had a problem with get user and team, request didnt work for me. With another file all was ok.
So I changed func get_slack_api_data in zerver/data_imports/slack.py
![image](https://user-images.githubusercontent.com/43493241/136049833-18563dba-7333-493f-bf0d-a40637ca092d.png)
Maybe it is better too 


Discussion:
Thanks for the report!  Can you give an example filename for the Slack export zip file to make it easier to reproduce?

@rht @hackerkid @garg3133 can you help investigate this?
Hello @zulip/server-misc, @zulip/server-production members, this issue was labeled with the "area: export/import", "area: production installer" labels, so you may want to check it out!

<!-- areaLabelAddition -->

[slack_export.zip](https://github.com/zulip/zulip/files/7291506/slack_export.zip)
@timabbott it is an empty zip with 2 folders, on this folders I had problem
Awesome, that should make this accessible for anyone to investigate and debug.  
If I read the OP correctly, using `ZipFile` solves the unicode problem?
Yeah.  It seems possible it could be solved by just calling `unzip` in some slightly modified way; I don't see a strong reason to prefer one approach over the other.
I think the problem is that the zip file is created on Windows, and hence it uses Windows encoding. Here is a test:
- First I extract using `7z x`, which produces the folders with the correct names, `stöcklin` and `відкритки`
- Then I make a zip file from a folder, using `zip` command

This newly created zip file will output a proper Unicode name. To prove it, here is a sample zip file: 
[відкритки.zip](https://github.com/zulip/zulip/files/7304195/default.zip).

Nevertheless, we should expect a Windows-encoded zip file, and hence should use that `zipfile` Python library.

Resolving patch:
diff --git a/zerver/data_import/slack.py b/zerver/data_import/slack.py
--- a/zerver/data_import/slack.py
+++ b/zerver/data_import/slack.py
@@ -4,6 +4,7 @@
 import secrets
 import shutil
 import subprocess
+import zipfile
 from collections import defaultdict
 from typing import Any, Dict, Iterator, List, Optional, Set, Tuple, Type, TypeVar
 
@@ -1290,7 +1291,8 @@ def do_convert_data(original_path: str, output_dir: str, token: str, threads: in
         if not os.path.exists(slack_data_dir):
             os.makedirs(slack_data_dir)
 
-        subprocess.check_call(["unzip", "-q", original_path, "-d", slack_data_dir])
+        with zipfile.ZipFile(original_path) as zipObj:
+            zipObj.extractall(slack_data_dir)
     elif os.path.isdir(original_path):
         slack_data_dir = original_path
     else:


Test patch:
Repository: zulip/zulip

Issue:
Usr stream subscription status, private streams, and realm admins
For private streams, the endpoint
```
/users/{user_id}/subscription/{stream_id}
```
will show invalid "stream id" even for administrators (I do get the stream id from `client.get_streams(include_all_active=True)`, so I should have access to the stream).

My impression is that it would be safe and reasonable to add `allow_realm_admin=True` to this call:

https://github.com/zulip/zulip/blob/706ec9714c76d98c2ac230c3a5adf35dfd0c438a/zerver/views/users.py#L636

and experimentally that fixes it.
So the fix is trivial, but before proposing a patch, I would need to look at creating tests that go with it, which I didn't do yet.



Discussion:
@t-vi thanks for the report!  I believe your proposed change is correct; sorry that we didn't see this report at the time.

@sahil839 do you want to just do a quick PR changing that and adding tests?  
Hello @zulip/server-api, @zulip/server-streams members, this issue was labeled with the "area: stream settings", "area: api" labels, so you may want to check it out!

<!-- areaLabelAddition -->


Resolving patch:
diff --git a/zerver/views/users.py b/zerver/views/users.py
--- a/zerver/views/users.py
+++ b/zerver/views/users.py
@@ -639,7 +639,7 @@ def get_subscription_backend(
     stream_id: int = REQ(json_validator=check_int, path_only=True),
 ) -> HttpResponse:
     target_user = access_user_by_id(user_profile, user_id, for_admin=False)
-    (stream, sub) = access_stream_by_id(user_profile, stream_id)
+    (stream, sub) = access_stream_by_id(user_profile, stream_id, allow_realm_admin=True)
 
     subscription_status = {"is_subscribed": subscribed_to_stream(target_user, stream_id)}
 


Test patch:
diff --git a/zerver/tests/test_users.py b/zerver/tests/test_users.py
--- a/zerver/tests/test_users.py
+++ b/zerver/tests/test_users.py
@@ -1320,6 +1320,26 @@ def test_get_user_subscription_status(self) -> None:
         )
         self.assertTrue(result["is_subscribed"])
 
+        self.login("iago")
+        stream = self.make_stream("private_stream", invite_only=True)
+        # Unsubscribed admin can check subscription status in a private stream.
+        result = orjson.loads(
+            self.client_get(f"/json/users/{iago.id}/subscriptions/{stream.id}").content
+        )
+        self.assertFalse(result["is_subscribed"])
+
+        # Unsubscribed non-admins cannot check subscription status in a private stream.
+        self.login("shiva")
+        result = self.client_get(f"/json/users/{iago.id}/subscriptions/{stream.id}")
+        self.assert_json_error(result, "Invalid stream id")
+
+        # Subscribed non-admins can check subscription status in a private stream
+        self.subscribe(self.example_user("shiva"), stream.name)
+        result = orjson.loads(
+            self.client_get(f"/json/users/{iago.id}/subscriptions/{stream.id}").content
+        )
+        self.assertFalse(result["is_subscribed"])
+
 
 class ActivateTest(ZulipTestCase):
     def test_basics(self) -> None:
Repository: zulip/zulip

Issue:
Add compose box button to insert local time
The [local times widget](https://zulip.com/help/format-your-message-using-markdown#mention-a-time) is very useful, but can be difficult to discover, and requires remembering a bit of syntax (`<time`) in order to use.

 To address these challenges, we should add a compose box button to insert a local time.

* Icon: https://fontawesome.com/v4.7/icon/clock-o
* Location: Between the smiley and `GIF` under the compose box
* Action: Opens the calendar widget (currently accessed via typing `<time`) for inserting a timezone-sensitive time. 

The time should be inserted in the position of the cursor.


Discussion:
Hello @zulip/server-compose members, this issue was labeled with the "area: compose" label, so you may want to check it out!

<!-- areaLabelAddition -->

@zulipbot claim
@zulipbot claim

Resolving patch:
diff --git a/tools/lib/capitalization.py b/tools/lib/capitalization.py
--- a/tools/lib/capitalization.py
+++ b/tools/lib/capitalization.py
@@ -105,6 +105,7 @@
         '<a href="/api" target="_blank">API</a> documentation?'
     ),
     r'Most stream administration is done on the <a href="/#streams">Streams page</a>.',
+    r"Add global time<br />Everyone sees global times in their own time zone.",
     r"one or more people...",
     r"confirmation email",
     r"invites remaining",


Test patch:
Repository: zulip/zulip

Issue:
Clear `prefers_web_public_view` cookie when logging in or registering an account
In #19902, we added functionality to remember who had accessed Zulip as a spectator and immediately take them to the spectator experience.  We probably want to clear the `prefers_web_public_view` cookie when logging in via `do_login` or registering a new account, since the user likely doesn't want to get the spectator experience on future visits to the site.


Discussion:
@timabbott is this issue open?
Howdy!  It's open, though it's part of a very large project and may require some detailed knowledge or good code reading skills, so if you're relatively new to reading large codebases, it may not be a good fit; that's why we haven't tagged the "help wanted" label on it.  

Resolving patch:
diff --git a/zerver/views/home.py b/zerver/views/home.py
--- a/zerver/views/home.py
+++ b/zerver/views/home.py
@@ -148,6 +148,10 @@ def home_real(request: HttpRequest) -> HttpResponse:
     if request.user.is_authenticated:
         user_profile = request.user
         realm = user_profile.realm
+
+        # User is logged in and hence no longer `prefers_web_public_view`.
+        if "prefers_web_public_view" in request.session.keys():
+            del request.session["prefers_web_public_view"]
     else:
         realm = get_valid_realm_from_request(request)
 


Test patch:
diff --git a/zerver/tests/test_home.py b/zerver/tests/test_home.py
--- a/zerver/tests/test_home.py
+++ b/zerver/tests/test_home.py
@@ -330,6 +330,12 @@ def test_logged_out_home(self) -> None:
         ]
         expected_keys = [i for i in self.expected_page_params_keys if i not in removed_keys]
         self.assertEqual(actual_keys, expected_keys)
+        self.assertEqual(self.client.session.get("prefers_web_public_view"), True)
+
+        # Web public session key should clear once user is logged in
+        self.login("hamlet")
+        self.client_get("/")
+        self.assertEqual(self.client.session.get("prefers_web_public_view"), None)
 
     def test_home_under_2fa_without_otp_device(self) -> None:
         with self.settings(TWO_FACTOR_AUTHENTICATION_ENABLED=True):
Repository: zulip/zulip

Issue:
Accepting an invitation from a now-deactivated user produces a server error
It doesn't show any errors to the user registering, happily.  But we should add a check in `process_new_human_user` that `prereg_user.referred_by` isn't deactivated before trying to send the message:
https://github.com/zulip/zulip/blob/c97956c5ff9457e01e642a8f0695e97b44921f3a/zerver/lib/actions.py#L481-L490



Discussion:
@alxmv Is it that a user is considered deactivated if the `is_active` attribute of the User is set to False?
Hello @zulip/server-onboarding members, this issue was labeled with the "area: invitations" label, so you may want to check it out!

<!-- areaLabelAddition -->

@Parth-Mittal-NITK: Yup!
@zulipbot claim
@alexmv Cool. 
Any manual tests required since you have mentioned that it doesn't show any error to the user registering?

Resolving patch:
diff --git a/zerver/lib/actions.py b/zerver/lib/actions.py
--- a/zerver/lib/actions.py
+++ b/zerver/lib/actions.py
@@ -503,7 +503,12 @@ def process_new_human_user(
     add_new_user_history(user_profile, streams)
 
     # mit_beta_users don't have a referred_by field
-    if not mit_beta_user and prereg_user is not None and prereg_user.referred_by is not None:
+    if (
+        not mit_beta_user
+        and prereg_user is not None
+        and prereg_user.referred_by is not None
+        and prereg_user.referred_by.is_active
+    ):
         # This is a cross-realm private message.
         with override_language(prereg_user.referred_by.default_language):
             internal_send_private_message(


Test patch:
diff --git a/zerver/tests/test_signup.py b/zerver/tests/test_signup.py
--- a/zerver/tests/test_signup.py
+++ b/zerver/tests/test_signup.py
@@ -1281,6 +1281,31 @@ def test_invite_mirror_dummy_user(self) -> None:
             inviter.email,
         )
 
+    def test_invite_from_now_deactivated_user(self) -> None:
+        """
+        While accepting an invitation from a user,
+        processing for a new user account will only
+        be completed if the inviter is not deactivated
+        after sending the invite.
+        """
+        inviter = self.example_user("hamlet")
+        self.login_user(inviter)
+        invitee = self.nonreg_email("alice")
+
+        result = self.invite(invitee, ["Denmark"])
+        self.assert_json_success(result)
+
+        prereg_user = PreregistrationUser.objects.get(email=invitee)
+        change_user_is_active(inviter, False)
+        do_create_user(
+            invitee,
+            "password",
+            inviter.realm,
+            "full name",
+            prereg_user=prereg_user,
+            acting_user=None,
+        )
+
     def test_successful_invite_user_as_owner_from_owner_account(self) -> None:
         self.login("desdemona")
         invitee = self.nonreg_email("alice")
Repository: zulip/zulip

Issue:
Display whether fields are nullable in the OpenAPI documentation
It appears in our `zulip.yaml` file that we have various fields, like the `stream_weekly_traffic` part of the [register response](https://zulip.com/api/register-queue), that are correctly marked in the OpenAPI field as `nullable: true`.  However, we don't display that fact when displaying the type as e.g. "integer" for one of these fields.  We should fix that.

See https://chat.zulip.org/#narrow/stream/378-api-design/topic/.22type.22.20annotations.20and.20.60null.60/near/1282157 for background and discussion on how to display the type.


Discussion:
Hello @zulip/server-api members, this issue was labeled with the "area: documentation (api and integrations)" label, so you may want to check it out!

<!-- areaLabelAddition -->


Resolving patch:
diff --git a/zerver/lib/markdown/api_arguments_table_generator.py b/zerver/lib/markdown/api_arguments_table_generator.py
--- a/zerver/lib/markdown/api_arguments_table_generator.py
+++ b/zerver/lib/markdown/api_arguments_table_generator.py
@@ -191,4 +191,6 @@ def generate_data_type(schema: Mapping[str, Any]) -> str:
         data_type = "(" + generate_data_type(schema["items"]) + ")[]"
     else:
         data_type = schema["type"]
+        if "nullable" in schema and schema["nullable"]:
+            data_type = data_type + " | null"
     return data_type


Test patch:
Repository: zulip/zulip

Issue:
Hide data about custom profile fields from logged out users
We do not display custom profile data in the logged out UI. To minimize the amount of information about users and organizations that is world-viewable, we should remove access to custom profile field data from the API for logged out users.


Discussion:
This should be a straightforward change to the `events.py` code; @amanagr FYI.  

(To be clear, this is just the data shared `events.py` about which custom profile fields exist in the organization; I think we already don't share users' values for the custom profile fields via code in `zerver/lib/users.py`).

Resolving patch:
diff --git a/zerver/lib/events.py b/zerver/lib/events.py
--- a/zerver/lib/events.py
+++ b/zerver/lib/events.py
@@ -143,7 +143,9 @@ def fetch_initial_state_data(
     if want("alert_words"):
         state["alert_words"] = [] if user_profile is None else user_alert_words(user_profile)
 
-    if want("custom_profile_fields"):
+    # Spectators can't access full user profiles or personal settings,
+    # so there's no need to send custom profile field data.
+    if want("custom_profile_fields") and user_profile is not None:
         fields = custom_profile_fields_for_realm(realm.id)
         state["custom_profile_fields"] = [f.as_dict() for f in fields]
         state["custom_profile_field_types"] = {
@@ -392,6 +394,8 @@ def fetch_initial_state_data(
             user_profile,
             client_gravatar=client_gravatar,
             user_avatar_url_field_optional=user_avatar_url_field_optional,
+            # Don't send custom profile field values to spectators.
+            include_custom_profile_fields=user_profile is not None,
         )
         state["cross_realm_bots"] = list(get_cross_realm_dicts())
 


Test patch:
diff --git a/zerver/tests/test_home.py b/zerver/tests/test_home.py
--- a/zerver/tests/test_home.py
+++ b/zerver/tests/test_home.py
@@ -349,6 +349,8 @@ def test_logged_out_home(self) -> None:
         page_params = self._get_page_params(result)
         actual_keys = sorted(str(k) for k in page_params.keys())
         removed_keys = [
+            "custom_profile_field_types",
+            "custom_profile_fields",
             "last_event_id",
             "narrow",
             "narrow_stream",