Repository: oppia/oppia

Issue:
Provide a way to automatically provide backend constants to the frontend as actual constants
Oppia has many constants defined in its backend domain files, including https://github.com/oppia/oppia/blob/develop/core/domain/rights_manager.py#L36 and https://github.com/oppia/oppia/blob/develop/core/domain/collection_domain.py#L32, among others. These constants must be reused in many places in the frontend. The mechanisms for getting these values from the backend to the frontend are less-than-ideal. Currently, it's done in one of three ways:
1. The value of the constant is hardcoded in places where it's needed in the frontend (highly prone to error if the constant values change in the backend; it also makes it hard to track down the constants)
2. The constant is passed in our `GLOBALS` object which shares major state information from the backend to the frontend (dangerous; elements in `GLOBALS` are not actually constants)
3. Hardcode the values in the frontend, but as Angular constants and inject their values (the least susceptible to changing, but still duplicates the values of the constants which could lead to potential skew)

This issue proposes a solution to the problem. All constants should be sent from the backend to the frontend (similar to mechanism (2) above), but without the susceptibility of being changeable (such as the benefits from (3) above). There needs to be a mechanism in the backend which allows backend constants to be automatically sent to the frontend and bound as Angular constants (similar to https://github.com/oppia/oppia/blob/develop/core/templates/dev/head/domain/collection/CollectionUpdateService.js#L25).

There are at least two approaches to doing this. One involves automatically binding all constants from the backend and sending them to the frontend. Although convenient, this method goes against our efforts to reduce our data sent across the wire. Another approach is allowing the backend controllers to specify which constants they wish to provide to the frontend. This is not quite as clean as the first solution; tightly coupling the backend controllers which constants arbitrarily used in the frontend is not necessarily ideal, either. The assignee of this issue should consider investigating other approaches to providing the above solution, perhaps with better drawbacks than these.



Discussion:
@BenHenning -- is it reasonable to have a json file shared between the f/e and b/e that contains all relevant constants? We might even be able to generate a new version of this file per release for caching, so that it doesn't need to be fetched separately each time.

@seanlip definitely another viable solution. That would help get rid of a lot of feconf, which might be nice.

Adding comments from #542 (a similar issue):

---

Some backend constants are passed into the frontend in a somewhat ad hoc way 
via a GLOBALS dict. This results in their having to be redeclared in karma 
tests, and also in frontend controllers reaching out to the global namespace to 
read from GLOBALS. Both of these seem suboptimal.

It would be nice to instead have a single data file that serves as a 'single 
source of truth' for all such constants that are shared between the frontend 
and the backend. I can think of a couple of ways this might work:

(a) have a JSON file containing constants; this is opened and read for each 
request and parsed into constants used by the backend. This seems rather 
labor-intensive, though.
(b) have feconf.py as we do now, but group some of the constants in it into a 
APP_CONSTANTS dict. Add a backend handler whose only job is to take this dict 
and serve it to the frontend as JSON.

(b) would obviate the need for GLOBALS, since it allows the constants to be 
fetched directly via an Angular service. However, it wouldn't fix the karma 
problem, because the karma tests don't have a backend server to issue requests 
to. Perhaps there is some other solution.

Hi @Tamakimouto -- this is a somewhat harder problem than the previous PRs you've tackled, since it is a bit open-ended, and the best solution isn't yet known. So, I think it might be a good challenge!

If you'd like to have a go at investigating this, here's what I would suggest as a first milestone: prepare a PR that establishes a pattern for sharing a _single_ constant of your choice between the frontend and the backend, and, if it's not self-evident, instructions for how to add further constants. The pattern should be replicable for other constants, and the value of the constant should be specified exactly once in the entire codebase. In addition:

- Adding further constants should be as simple an operation as possible. Ideally, a dev would just need to add the constant's name and value to a single place in one file.
- The dev workflow should not be changed; there should be no need for a 'build' step to generate constants for the frontend.
- It's fine if other workflows need changing. For example, maybe we may need to build a test-constants file before running the karma unit tests. That is fine; we can just work it into that workflow.
- For now, don't worry about issues with sending unneeded constants over. Once we have a pattern, we can think subsequently about bundling as a separate step, so that we don't send more stuff than necessary. But for now I think it is fine to assume that we have a big bunch of constants that just need to be shared between backend and frontend.

As you investigate possible approaches, please add notes to this issue to explain paths you've explored (and perhaps rejected), since that saves others from having to reinvent the wheel and helps provide a reference for why the eventual solution is the way it is.

Thanks!
Hi @Tamakimouto -- how is this going?
Hi, I have been looking into this a little bit. And I have one questions.
> have a JSON file containing constants; this is opened and read for each
request and parsed into constants used by the backend. This seems rather
labor-intensive, though.

This approach is suggesting, that the constants.json is opened by backend in some handler, some part of constants is then sent to frontend and converted to Angular constants?
I think so, yeah. But I'm not sure it's a good approach, and actually it's probably better to just serve the constants.json file statically, like we do for the json files in assets/i18n/*.json. (At the time that comment was written, I don't think we had the i18n pattern in place yet.)
So, after some trying, I think the best approach is to use the JSON file, though the file must be loaded synchronously in vanilla JS and then passed to Angular constant similarly to GLOBALS. The file should be cached so there is not some huge data transfer. Also, the advantage is that the usage of constants is same in frontend and backend.
I only need to solve the issue with karma tests, because the variable where the JSON data are loaded with vanillaJS isn't visible in app.js in karma tests, so it can't be passed to angular.constant().
Thanks for the update, @vojtechjelinek!

Not sure if the following solves your remaining issue (I'm writing this in a hurry) but you might want to take a look at https://github.com/oppia/oppia/blob/develop/core/tests/karma-globals.js 
OK, I've had time to read this a bit more carefully. Apart from the above link, I also remember doing something with JSON files in karma before. See:

https://github.com/oppia/oppia/commit/beabba499665b156a99b72b003cbc8d1ec4c19d8#diff-51f08e56ffff03cdfaf5d26933eca640R34

(Note also https://github.com/oppia/oppia/commit/beabba499665b156a99b72b003cbc8d1ec4c19d8#diff-90fe206654dfa04eda2d95a73cc44fdfR74 and other changes in karma.conf.js.)

Again, not sure if this is useful -- I don't know if it's possible to retrieve the JSON file "in time" to put it in an angular constant. If none of these approaches work then we'll have to find some other solution.

Resolving patch:
diff --git a/constants.py b/constants.py
new file mode 100644
--- /dev/null
+++ b/constants.py
@@ -0,0 +1,27 @@
+# coding: utf-8
+#
+# Copyright 2017 The Oppia Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS-IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Loads constants for backend use."""
+
+import json
+import os
+
+class Constants(dict):
+    """Transforms dict to object, attributes can be accesed by dot notation"""
+    __getattr__ = dict.__getitem__
+
+with open(os.path.join('assets', 'constants.json'), 'r') as f:
+    constants = Constants(json.load(f)) #pylint: disable=invalid-name
diff --git a/core/controllers/editor.py b/core/controllers/editor.py
--- a/core/controllers/editor.py
+++ b/core/controllers/editor.py
@@ -22,6 +22,7 @@
 
 import jinja2
 
+from constants import constants
 from core.controllers import base
 from core.domain import config_domain
 from core.domain import dependency_registry
@@ -160,7 +161,7 @@ class ExplorationPage(EditorHandler):
 
     def get(self, exploration_id):
         """Handles GET requests."""
-        if exploration_id in feconf.DISABLED_EXPLORATION_IDS:
+        if exploration_id in constants.DISABLED_EXPLORATION_IDS:
             self.render_template(
                 'pages/error/disabled_exploration.html',
                 iframe_restriction=None)


Test patch:
diff --git a/constants_test.py b/constants_test.py
new file mode 100644
--- /dev/null
+++ b/constants_test.py
@@ -0,0 +1,32 @@
+# Copyright 2017 The Oppia Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS-IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for Constants object and cosntants.json file."""
+
+import os
+import json
+
+from core.tests import test_utils #pylint: disable=relative-import
+
+class ConstantsTests(test_utils.GenericTestBase):
+
+    def test_constants_file_is_existing(self):
+        """Test if the constants file is existing."""
+        self.assertTrue(os.path.isfile(os.path.join(
+            'assets', 'constants.json')))
+
+    def test_constants_file_is_valid_json(self):
+        """Test if the constants file is valid json file."""
+        with open(os.path.join('assets', 'constants.json'), 'r') as f:
+            self.assertTrue(isinstance(json.load(f), dict))
diff --git a/core/tests/karma-constants.js b/core/tests/karma-constants.js
new file mode 100644
--- /dev/null
+++ b/core/tests/karma-constants.js
@@ -0,0 +1,27 @@
+// Copyright 2017 The Oppia Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS-IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @fileoverview Loads constants for the Karma test environment.
+ */
+
+/* This should be kept in parity with how CONSTANTS is set in base.html */
+var constants = (function() {
+  var request = new XMLHttpRequest();
+  request.overrideMimeType('application/json');
+  request.open(
+   'GET', GLOBALS.ASSET_DIR_PREFIX + '/assets/constants.json', false);
+  request.send(null);
+  return JSON.parse(request.responseText);
+})();
diff --git a/core/tests/karma-globals.js b/core/tests/karma-globals.js
--- a/core/tests/karma-globals.js
+++ b/core/tests/karma-globals.js
@@ -26,7 +26,8 @@ var GLOBALS = {
   }, {
     id: 'en',
     text: 'English'
-  }]
+  }],
+  ASSET_DIR_PREFIX: ''
 };
 
 /* This function overwrites the translationProvider for a dummy function
diff --git a/core/tests/karma.conf.js b/core/tests/karma.conf.js
--- a/core/tests/karma.conf.js
+++ b/core/tests/karma.conf.js
@@ -11,6 +11,12 @@ module.exports = function(config) {
     frameworks: ['jasmine'],
     files: [
       'core/tests/karma-globals.js',
+      // Constants must be loaded before everything else.
+      {
+        pattern: 'assets/constants.json',
+        included: false
+      },
+      'core/tests/karma-constants.js',
       // Since jquery,jquery-ui,angular,angular-mocks and math-expressions
       // are not bundled, they will be treated separately.
       'third_party/static/jquery-3.0.0/jquery.min.js',
@@ -39,6 +45,12 @@ module.exports = function(config) {
       'core/templates/dev/head/**/*-e2e.js',
       'extensions/**/protractor.js'
     ],
+    proxies: {
+      // Karma serves files under the /base directory.
+      // We need to access files in assets folder, without modifying the code,
+      // so we need to proxy the requests from /assets/ to /base/assets/.
+      '/assets/': '/base/assets/'
+    },
     preprocessors: {
       'core/templates/dev/head/*.js': ['coverage'],
       // When all controllers were converted from global functions into the
diff --git a/core/tests/protractor/cacheSlugs.js b/core/tests/protractor/cacheSlugs.js
--- a/core/tests/protractor/cacheSlugs.js
+++ b/core/tests/protractor/cacheSlugs.js
@@ -56,7 +56,11 @@ describe('Cache Slugs', function() {
   it('should check that errors get logged for missing resources', function() {
     browser.get(ERROR_PAGE_URL_SUFFIX);
     var expectedErrors = [
-      'http://localhost:9001/build/fail/logo/288x128_logo_white.png'
+      'http://localhost:9001/build/fail/logo/288x128_logo_white.png',
+      // Warning fired by synchronous AJAX request in
+      // core/templates/dev/head/pages/header_js_libs.html.
+      // This warning is expected and it's needed for constants loading.
+      'Synchronous XMLHttpRequest on the main thread'
     ];
     checkConsoleErrorsExist(expectedErrors);
   });