Repository: oppia/oppia

Issue:
Create a python script that does a size check for third_party.
This is a suggestion by @hoangviet1993 (thanks @hoangviet1993!): 

Let's include a python script that does a size-check (i.e. number of files) for third_party. Should It get above , let's say 7k?, the Travis build should fail.

The aim of this is to prevent us accidentally breaching the 10k file limit on App Engine. See also #5201 which is related.

/cc @vojtechjelinek 


Discussion:
Also /cc @apb7 since this relates to dev workflow.
Note: this is meant to be a count of files, not the size of the files.
Hi, is anyone working in this issue? If not, I would like to work on it. But I have some questions about the task. The explanation says that the script causes a crash in Travis build when there are more than 7k files, I was looking at the others scripts that make Travis build crash and they have a shell script that runs the python script and if it fails then the scripts fails and finally the .travis.yaml builds fails. My question is, the task includes creating that shell scrips and the code necessary in the .travis.yaml or only the python script? Or is there any other way of making the Travis build fail using only the python script? And also, does directories count or just the files?
Hi @tonadev, go for it!

The task includes creating whatever code is needed to achieve the desired result -- i.e. if we get to more than a certain file count then Travis should fail.

Also, only the files count, not the directories.

Thanks! 
I think you can include the check in `install_third_party.py`, after `_install_third_party_libs()` is run. Does it make sense? (or just run a separate python script after `install_third_party.py` has finished installing is fine too)
Hi @tonadev, there is a Travis build that runs on every PR. The build for that PR should catch the issue and prevent that PR from being merged (right?) So there shouldn't be a cascading effect on other PRs...

We could also just make this a regular presubmit check, but only if it doesn't take a long time, otherwise it slows down folks a lot.

I'm not sure whether including the check in install_third_party.py suffices though. The developer may just miss the error message...
I realized about that I was wrong, that why supposedly  I deleted the comment.

Resolving patch:
diff --git a/scripts/third_party_size_check.py b/scripts/third_party_size_check.py
new file mode 100644
--- /dev/null
+++ b/scripts/third_party_size_check.py
@@ -0,0 +1,77 @@
+# coding: utf-8
+#
+# Copyright 2018 The Oppia Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS-IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Script that runs a size check in third-party folder and errors if
+size limit is exceeded. The aim of this is to prevent us accidentally
+breaching the 10k file limit on App Engine.
+"""
+
+import os
+import sys
+
+THIRD_PARTY_PATH = os.path.join(os.getcwd(), 'third_party')
+THIRD_PARTY_SIZE_LIMIT = 7000
+
+
+def _check_size_in_dir(dir_path):
+    """Recursive method that checks the number of files inside the given
+    directory.
+
+    Args:
+         dir_path. str. The directory which files will be counted.
+
+    Returns:
+        The number of files inside the given directory.
+    """
+    number_of_files_in_dir = 0
+    for name in os.listdir(dir_path):
+        if os.path.isfile(os.path.join(dir_path, name)):
+            number_of_files_in_dir += 1
+        else:
+            if os.path.isdir(os.path.join(dir_path, name)):
+                number_of_files_in_dir += _check_size_in_dir(
+                    os.path.join(dir_path, name))
+    return number_of_files_in_dir
+
+
+def _check_third_party_size():
+    """Checks if the third-party size limit has been exceeded."""
+    number_of_files_in_third_party = _check_size_in_dir(THIRD_PARTY_PATH)
+    print ''
+    print '------------------------------------------------------'
+    print '    Number of files in third-party folder: %d' % (
+        number_of_files_in_third_party)
+    print ''
+    if number_of_files_in_third_party > THIRD_PARTY_SIZE_LIMIT:
+        print(
+            '    ERROR: The third-party folder size exceeded the %d files'
+            ' limit.' % THIRD_PARTY_SIZE_LIMIT)
+        print '------------------------------------------------------'
+        print ''
+        sys.exit(1)
+    else:
+        print '    The size of third-party folder is within the limits.'
+        print '------------------------------------------------------'
+        print ''
+        print 'Done!'
+        print ''
+
+
+if __name__ == '__main__':
+    print 'Running third-party size check'
+    _check_third_party_size()
+    print 'Third-party folder size check passed.'
+    print ''


Test patch: