Repository: zulip/zulip

Issue:
Make hard-coded `jitsi_server_url` and friends accessible by `/register`
In the [Events subsystem doc](https://zulip.readthedocs.io/en/latest/subsystems/events-system.html), `page_params` is documented as the way to access data provided in the `/register` response.

I learned recently (@showell points out [here](https://chat.zulip.org/#narrow/stream/127-integrations/topic/Jitsi.20Videocalls/near/829298)) that some fields on `page_params` do not come from the `/register` response. `page_params` is initialized with data from the server settings via the template at templates/zerver/base.html — the fields are specified at line ~225 of zerver/views/home.py — so these fields are essentially hard-coded for use by the web app only.

At least a few of these will be nice to have for the mobile app. I consider https://github.com/zulip/zulip-mobile/issues/3546 to be blocked on this, since `jitsi_server_url` is one of these, but there are other ones that it seems helpful for mobile to be aware of, such as `max_file_upload_size` (I don't think we're currently limiting uploads by file size).

If some of these could instead be made available through `/register`, the mobile app could easily consume them. (Perhaps they can all be treated this way, but I don't know the intricacies of each.)


Discussion:
Yeah, the intent is to move all of the ones that are not webapp-specific to the `realm` section of the `/register` response; we did most of them a couple years ago and then forgot to finish, I think.

It's a very simple change to move one of these; we should probably just figure out the fully list we want to move and move them all.

Some, like `TEST_SUITE`, are likely meaningless; let me see if I can put together a proposal.
One thing to discuss is naming; we possible want to name these as e.g. `settings_root_domain_uri` or `server_root_domain_uri` to distinguish them.  And also what `event_types` one needs to request to get them; by default I'd do `realm`.  But here's my summary.

These seem useful and could be migrated over modulo potentially adding a naming prefix:
        development_environment = settings.DEVELOPMENT,                                              
        max_file_upload_size  = settings.MAX_FILE_UPLOAD_SIZE,                                       
        max_avatar_file_size  = settings.MAX_AVATAR_FILE_SIZE,                                       
        server_generation     = settings.SERVER_GENERATION,                                          
        server_inline_image_preview = settings.INLINE_IMAGE_PREVIEW,                                 
        server_inline_url_embed_preview = settings.INLINE_URL_EMBED_PREVIEW,                         
        password_min_length = settings.PASSWORD_MIN_LENGTH,                                          
        password_min_guesses  = settings.PASSWORD_MIN_GUESSES,                                       
        jitsi_server_url      = settings.JITSI_SERVER_URL,                                           
        server_avatar_changes_disabled = settings.AVATAR_CHANGES_DISABLED,                           
        server_name_changes_disabled = settings.NAME_CHANGES_DISABLED,                               

This is for casper tests or the web UI only.
        test_suite            = settings.TEST_SUITE,                                                 
        insecure_desktop_app  = is_outdated_desktop_app(request.META.get("HTTP_USER_AGENT", "")),    
        login_page            = settings.HOME_NOT_LOGGED_IN,                                         
        save_stacktraces      = settings.SAVE_FRONTEND_STACKTRACES,                                  
        warn_no_email         = settings.WARN_NO_EMAIL,                                              
        search_pills_enabled  = settings.SEARCH_PILLS_ENABLED,                                       



These are unused and should perhaps be removed:
        has_mobile_devices    = user_profile is not None and num_push_devices_for_user(user_profile) > 0,                                                                                                
        prompt_for_invites    = prompt_for_invites,                                                  
        root_domain_uri       = settings.ROOT_DOMAIN_URI,                                            


These are data about the set of options used for user/organization settings.  They're things we ultimately want to be part of the API, but perhaps don't make sense to move now as they may not be in a reusable form.
        bot_types             = get_bot_types(user_profile),                                         
        default_language_name = get_language_name(register_ret['default_language']),                 
        language_list         = get_language_list(),                                                 

This may be in too specific a format to be reusable?  Needs checking.
        language_list_dbl_col = get_language_list_for_templates(register_ret['default_language']),   

These seem webapp-specific and/or things we might remove/refactor before reusing:
        needs_tutorial        = needs_tutorial,                                                      
        two_fa_enabled        = two_fa_enabled,                                                      
        two_fa_enabled_user   = two_fa_enabled and bool(default_device(user_profile)),               
        # Actually is whether the realm has only one user.
        first_in_realm        = first_in_realm,                                                      
        initial_servertime    = time.time(),  # Used for calculating relative presence age           

I think there might be an active PR that removes these:
        have_initial_messages = user_has_messages,                                                   
        furthest_read_time    = furthest_read_time,                                                  

I'm not sure what the story with this is but I think it's mostly unused:
        poll_timeout          = settings.POLL_TIMEOUT,                                               


Thanks, Tim!

After having posted this, I remembered that `/server_settings` exists ([doc](https://zulipchat.com/api/server-settings)); I'd been using that recently for the zulip_version in https://github.com/zulip/zulip-mobile/pull/3839.

Might some of these belong there, rather than in the /register response?
I think basically no.  `/server_settings` is intended to be just the information for clients needed to determine how to connect to and authenticate with a Zulip server; I think we want to keep that interface at the minimum required to achieve that goal.  

Further, I think it should be OK for an already-authenticated client to only fetch `/server_settings` when it wants to authenticate and setup an API key; any details that are important after that point should ideally be included in the `/register` response, which we expect clients to request more frequently in any plausible design.


Thanks!

For specific naming choices, Tim started [a chat thread](https://chat.zulip.org/#narrow/stream/3-backend/topic/.2Fregister.20interface/near/830111); good idea. I replied there.


> I think it should be OK for an already-authenticated client to only fetch `/server_settings` when it wants to authenticate and setup an API key; any details that are important after that point should ideally be included in the `/register` response, which we expect clients to request more frequently in any plausible design.

This discussion gives me the thought that perhaps `zulip_version` should appear in the `/register` response in addition to `/server_settings`. In https://github.com/zulip/zulip-mobile/pull/3839 we started making a `/server_settings` concurrently with every `/register`, for basically the same reason you might for like `jitsi_server_url` -- it may change from time to time. If `/register` had that data, we could go back to the behavior you describe. (... Heh, only for new servers, I guess. But in the long run that's all servers.)

(It still needs to be in `/server_settings`, because it can potentially matter already while trying to log in.)

Following that chat thread, here's a set of names for the first wave of these.

```
max_file_upload_size = settings.MAX_FILE_UPLOAD_SIZE,
max_avatar_file_size = settings.MAX_AVATAR_FILE_SIZE,
```

These two should gain a suffix `_mib` to clarify the units. (The docs say "megabytes" or "MB", but client software needs to know the actual meaning, which is in mebibytes.)


```
development_environment = settings.DEVELOPMENT,
server_generation = settings.SERVER_GENERATION,
password_min_length = settings.PASSWORD_MIN_LENGTH,
password_min_guesses = settings.PASSWORD_MIN_GUESSES,
jitsi_server_url = settings.JITSI_SERVER_URL,
server_inline_image_preview = settings.INLINE_IMAGE_PREVIEW,
server_inline_url_embed_preview = settings.INLINE_URL_EMBED_PREVIEW,
server_avatar_changes_disabled = settings.AVATAR_CHANGES_DISABLED,
server_name_changes_disabled = settings.NAME_CHANGES_DISABLED,
```

These other names are good, and can go into `/register` with the same name.

---

Then, a note on API naming choices, which is background for some of the choices above:

```
max_file_upload_size = settings.MAX_FILE_UPLOAD_SIZE,
max_avatar_file_size = settings.MAX_AVATAR_FILE_SIZE,
password_min_length = settings.PASSWORD_MIN_LENGTH,
password_min_guesses = settings.PASSWORD_MIN_GUESSES,
jitsi_server_url = settings.JITSI_SERVER_URL,
```

These are all pieces of server-level configuration, where in the future we potentially might (more likely in some cases than others) add realm/org-level settings to control them. The server-level setting would then be a default, or a maximum or minimum, for the realm/org-level setting.

If we do, the intended semantics is that these existing names continue to refer to *the setting that's actually in effect*. In particular, except for the org settings UI itself, any time a client might care about the value of these settings, it should be able to simply keep consulting these names.

For example, the `Realm` model might gain a setting for `password_min_length`. The server would then enforce that a chosen password's length is at least
```
max(settings.PASSWORD_MIN_LENGTH, realm.password_min_length)
```
and it would accordingly provide the client that value, like
```
password_min_length = max(settings.PASSWORD_MIN_LENGTH, realm.password_min_length),
```

Separately, for the benefit of specifically the org settings UI, we'd add a field to tell that UI what limits it should apply. This might look like
```
server_password_min_length = settings.PASSWORD_MIN_LENGTH,
```

This plan is different from what we've done with some settings in the past: instead of `foo` for the setting that's in effect and `server_foo` for the server-level limits relevant to the org settings UI, we've had `realm_foo` for the setting chosen by org admins and `server_foo` for the server-level setting. Two reasons we prefer the new plan are:
* The old pattern tends to require client code to manually combine the two settings, duplicating logic in the server.
* The old pattern means the name of the setting, and/or the client logic for applying it, changes when we add an org-level setting for an existing server-level setting. With the new pattern, the only client-side change needed is to implement the new fragment of org settings UI for the new org-level setting. In particular, clients that don't have an org settings UI don't have to change at all.



Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -29,7 +29,7 @@
 #
 # Changes should be accompanied by documentation explaining what the
 # new level means in templates/zerver/api/changelog.md.
-API_FEATURE_LEVEL = 3
+API_FEATURE_LEVEL = 4
 
 # Bump the minor PROVISION_VERSION to indicate that folks should provision
 # only when going from an old version of the code to a newer version. Bump
diff --git a/zerver/lib/events.py b/zerver/lib/events.py
--- a/zerver/lib/events.py
+++ b/zerver/lib/events.py
@@ -170,6 +170,17 @@ def fetch_initial_state_data(user_profile: UserProfile,
         state['plan_includes_wide_organization_logo'] = realm.plan_type != Realm.LIMITED
         state['upgrade_text_for_wide_organization_logo'] = str(Realm.UPGRADE_TEXT_STANDARD)
         state['realm_default_external_accounts'] = DEFAULT_EXTERNAL_ACCOUNTS
+        state['jitsi_server_url']                = settings.JITSI_SERVER_URL
+        state['development_environment']         = settings.DEVELOPMENT
+        state['server_generation']               = settings.SERVER_GENERATION
+        state['password_min_length']             = settings.PASSWORD_MIN_LENGTH
+        state['password_min_guesses']            = settings.PASSWORD_MIN_GUESSES
+        state['max_file_upload_size_mib']        = settings.MAX_FILE_UPLOAD_SIZE
+        state['max_avatar_file_size_mib']        = settings.MAX_AVATAR_FILE_SIZE
+        state['server_inline_image_preview']     = settings.INLINE_IMAGE_PREVIEW
+        state['server_inline_url_embed_preview'] = settings.INLINE_URL_EMBED_PREVIEW
+        state['server_avatar_changes_disabled']  = settings.AVATAR_CHANGES_DISABLED
+        state['server_name_changes_disabled']    = settings.NAME_CHANGES_DISABLED
 
         if realm.notifications_stream and not realm.notifications_stream.deactivated:
             notifications_stream = realm.notifications_stream
diff --git a/zerver/views/home.py b/zerver/views/home.py
--- a/zerver/views/home.py
+++ b/zerver/views/home.py
@@ -246,26 +246,15 @@ def home_real(request: HttpRequest) -> HttpResponse:
     # These end up in a global JavaScript Object named 'page_params'.
     page_params = dict(
         # Server settings.
-        development_environment = settings.DEVELOPMENT,
-        debug_mode            = settings.DEBUG,
-        test_suite            = settings.TEST_SUITE,
-        poll_timeout          = settings.POLL_TIMEOUT,
-        insecure_desktop_app  = insecure_desktop_app,
-        login_page            = settings.HOME_NOT_LOGGED_IN,
-        root_domain_uri       = settings.ROOT_DOMAIN_URI,
-        max_file_upload_size  = settings.MAX_FILE_UPLOAD_SIZE,
-        max_avatar_file_size  = settings.MAX_AVATAR_FILE_SIZE,
-        server_generation     = settings.SERVER_GENERATION,
-        save_stacktraces      = settings.SAVE_FRONTEND_STACKTRACES,
-        warn_no_email         = settings.WARN_NO_EMAIL,
-        server_inline_image_preview = settings.INLINE_IMAGE_PREVIEW,
-        server_inline_url_embed_preview = settings.INLINE_URL_EMBED_PREVIEW,
-        password_min_length = settings.PASSWORD_MIN_LENGTH,
-        password_min_guesses  = settings.PASSWORD_MIN_GUESSES,
-        jitsi_server_url      = settings.JITSI_SERVER_URL,
-        search_pills_enabled  = settings.SEARCH_PILLS_ENABLED,
-        server_avatar_changes_disabled = settings.AVATAR_CHANGES_DISABLED,
-        server_name_changes_disabled = settings.NAME_CHANGES_DISABLED,
+        debug_mode                      = settings.DEBUG,
+        test_suite                      = settings.TEST_SUITE,
+        poll_timeout                    = settings.POLL_TIMEOUT,
+        insecure_desktop_app            = insecure_desktop_app,
+        login_page                      = settings.HOME_NOT_LOGGED_IN,
+        root_domain_uri                 = settings.ROOT_DOMAIN_URI,
+        save_stacktraces                = settings.SAVE_FRONTEND_STACKTRACES,
+        warn_no_email                   = settings.WARN_NO_EMAIL,
+        search_pills_enabled            = settings.SEARCH_PILLS_ENABLED,
 
         # Misc. extra data.
         have_initial_messages = user_has_messages,
@@ -358,7 +347,7 @@ def home_real(request: HttpRequest) -> HttpResponse:
                                'show_webathena': show_webathena,
                                'embedded': narrow_stream is not None,
                                'invite_as': PreregistrationUser.INVITE_AS,
-                               'max_file_upload_size': settings.MAX_FILE_UPLOAD_SIZE,
+                               'max_file_upload_size_mib': settings.MAX_FILE_UPLOAD_SIZE,
                                },)
     patch_cache_control(response, no_cache=True, no_store=True, must_revalidate=True)
     return response


Test patch:
diff --git a/frontend_tests/node_tests/compose.js b/frontend_tests/node_tests/compose.js
--- a/frontend_tests/node_tests/compose.js
+++ b/frontend_tests/node_tests/compose.js
@@ -922,7 +922,7 @@ run_test('initialize', () => {
     global.document = 'document-stub';
     global.csrf_token = 'fake-csrf-token';
 
-    page_params.max_file_upload_size = 512;
+    page_params.max_file_upload_size_mib = 512;
 
     let setup_upload_called = false;
     let uppy_cancel_all_called = false;
diff --git a/frontend_tests/node_tests/upload.js b/frontend_tests/node_tests/upload.js
--- a/frontend_tests/node_tests/upload.js
+++ b/frontend_tests/node_tests/upload.js
@@ -176,7 +176,7 @@ run_test('upload_files', () => {
     upload.upload_files(uppy, config, []);
     assert.equal($("#compose-send-button").attr("disabled"), false);
 
-    page_params.max_file_upload_size = 0;
+    page_params.max_file_upload_size_mib = 0;
     let show_error_message_called = false;
     upload.show_error_message = (config, message) => {
         show_error_message_called = true;
@@ -186,7 +186,7 @@ run_test('upload_files', () => {
     upload.upload_files(uppy, config, files);
     assert(show_error_message_called);
 
-    page_params.max_file_upload_size = 25;
+    page_params.max_file_upload_size_mib = 25;
     let on_click_close_button_callback;
     $(".compose-send-status-close").one = (event, callback) => {
         assert.equal(event, "click");
diff --git a/zerver/tests/test_home.py b/zerver/tests/test_home.py
--- a/zerver/tests/test_home.py
+++ b/zerver/tests/test_home.py
@@ -100,8 +100,8 @@ def test_home(self) -> None:
             "last_event_id",
             "left_side_userlist",
             "login_page",
-            "max_avatar_file_size",
-            "max_file_upload_size",
+            "max_avatar_file_size_mib",
+            "max_file_upload_size_mib",
             "max_icon_file_size",
             "max_logo_file_size",
             "max_message_id",