Repository: oppia/oppia

Issue:
Linter Checking file that does not exist
<!--
  - Thanks for taking the time to report a bug in the Oppia project.
  - Before filing a new issue, please do a quick search to check that it hasn't
  - already been filed on the [issue tracker](https://github.com/oppia/oppia/issues)._
  -->

**Describe the bug**
Linter fails on attempting to lint on a file that does not exist. Happens both locally on computer and remotely on circleci


1. On CircleCI
```

Unable to run the complete lint test, please check the following stack trace and fix the errors:
+--------------------------+
Traceback (most recent call last):
  File "scripts/concurrent_task_utils.py", line 61, in run
    self.output = self.func()
  File "scripts/linters/js_ts_linter.py", line 860, in perform_all_lint_checks
    self.parsed_js_and_ts_files = self._validate_and_parse_js_and_ts_files()
  File "scripts/linters/js_ts_linter.py", line 185, in _validate_and_parse_js_and_ts_files
    file_content = FILE_CACHE.read(compiled_js_filepath)
  File "/home/circleci/oppia/scripts/linters/pre_commit_linter.py", line 168, in read
    return self._get_data(filepath, mode)[0]
  File "/home/circleci/oppia/scripts/linters/pre_commit_linter.py", line 199, in _get_data
    with python_utils.open_file(filepath, mode) as f:
  File "python_utils.py", line 106, in open_file
    raise IOError('Unable to open file: %s' % filename)
IOError: Unable to open file: /home/circleci/oppia/tmpcompiledjs8IV6JQ/translation-language-select.component.js
```

2. Locally (it seems to happen on a different file)
```
Unable to run the complete lint test, please check the following stack trace and fix the errors:
+--------------------------+
Traceback (most recent call last):
  File "scripts/concurrent_task_utils.py", line 61, in run
    self.output = self.func()
  File "scripts/linters/js_ts_linter.py", line 862, in perform_all_lint_checks
    self.parsed_js_and_ts_files = self._validate_and_parse_js_and_ts_files()
  File "scripts/linters/js_ts_linter.py", line 182, in _validate_and_parse_js_and_ts_files
    file_content = FILE_CACHE.read(compiled_js_filepath)
  File "/Users/davidchenggoogle/opensource/oppia/scripts/linters/pre_commit_linter.py", line 167, in read
    return self._get_data(filepath, mode)[0]
  File "/Users/davidchenggoogle/opensource/oppia/scripts/linters/pre_commit_linter.py", line 198, in _get_data
    with python_utils.open_file(filepath, mode) as f:
  File "python_utils.py", line 106, in open_file
    raise IOError('Unable to open file: %s' % filename)
IOError: Unable to open file: /Users/davidchenggoogle/opensource/oppia/tmpcompiledjsRjCxNX/translation-language-select.component.spec.js
```

**To Reproduce**
Steps to reproduce the behavior:
 1. Go to PR #9443
 2. Run linters

**Observed behavior**
A clear and concise description of what you have observed.
 1. Linter fails trying to access `oppia/tmpcompiledjs8IV6JQ/translation-language-select.component.js`

**Expected behavior**
A clear and concise description of what you expected to happen.
 1. Linter does not check file that does not exist



**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: MacOS 10.15.5




Discussion:
Assigning based on codeowners file
@kevinlee12 I tried to solve this problem but didn't find a solution, the file in this issue is created [here](https://github.com/oppia/oppia/blob/e764a00a32af4d4403eb23c72b5dc85350d88aaa/scripts/linters/js_ts_linter.py#L181) but there are problems in opening it. Any idea why this is happening?
Can you check the permissions of the file? 
@kevinlee12 Sorry but I don't get what do you mean by permissions.
This is as in if the current user has access to the file (ie, can you open
it manually?)

On Thu, Jun 11, 2020 at 11:09 PM Anshul Hudda <notifications@github.com>
wrote:

> @kevinlee12 <https://github.com/kevinlee12> Sorry but I don't get what do
> you mean by permissions.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/oppia/oppia/issues/9458#issuecomment-643085057>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABL52MGISS355NBXZI24QCDRWHBCJANCNFSM4NYQ7GVQ>
> .
>

@kevinlee12 The file is not being created at all. So I can't check the permission. I am having problems with this issue, is it possible if you take over this issue? Thanks!
sure.

On Mon, Jun 15, 2020 at 10:51 AM Anshul Hudda <notifications@github.com>
wrote:

> @kevinlee12 <https://github.com/kevinlee12> The file is not being created
> at all. So I can't check the permission. I am having problems with this
> issue, it would be great if you take over this issue. Thanks!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/oppia/oppia/issues/9458#issuecomment-644279057>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABL52MA6AGKLKZGLVV55NV3RWZNSVANCNFSM4NYQ7GVQ>
> .
>

Hi all, I've identified the problem but not sure how to proceed:

The file path returned from _compile_ts_file is `oppia/tmpcompiledjsXXXXXX/translation-language-select.component.js`, but the files generated by _compile_ts_file have a different structure:

![Screen Shot 2020-06-17 at 5 24 22 PM](https://user-images.githubusercontent.com/33111844/84952920-bf6ca800-b0c0-11ea-9565-2e5a140172ff.png)

As you can see, it should be returning `oppia/tmpcompiledjsXXXXXX/translation-language-select/translation-language-select.component.js`

Any idea why this may be? It seems the `tsc` command is generating unwanted sub folders, even though it is not specified in the command:
```
./node_modules/typescript/bin/tsc -outDir /Users/davidchenggoogle/opensource/oppia/tmpcompiledjsTmij8N -allowJS true -lib es2017,dom -noImplicitUseStrict true -skipLibCheck true -target es5 -typeRoots ./node_modules/@types /Users/davidchenggoogle/opensource/oppia/core/templates/pages/community-dashboard-page/translation-language-select/translation-language-select.component.ts typings/*
``` 

@kevinlee12 @Hudda 
thanks for finding the issue. 

 @ankita240796 @vojtechjelinek would any of you know?
I can't find out why this is happening. @Hudda But looking through the code I think that instead of compiling the files separately, I think we should compile all at once and then work with these. That way we can use `tsconfig.json` for the compilation and be a bit closer to the regular compile procedure that we use.
Looking at it more I found that when I mimick the `_compile_ts_file` the problem seems to arise from a fact that not only the `translation-language-select.component.js` is compiled but also `contribution-opportunities-backend-api.service.js`. It just gives me the confirmation that the separate compilation of ts files for linting is wrong, but can't really think of any quick fix for this. 
![image](https://user-images.githubusercontent.com/10142938/85005591-51c08a80-b159-11ea-9fa8-b3cf582aada8.png)

Just a note -- I chatted briefly with @vojtechjelinek and he suggests compiling all the files (like we do in the regular build process) instead of doing separate compilation. Maybe try that and see if it fixes the issue?
@vojtechjelinek also suggests: "Maybe try it using the tsconfig.json as the configuration -- I think that should work. And then retrieve the files from the file structure."

Resolving patch:
diff --git a/scripts/linters/general_purpose_linter.py b/scripts/linters/general_purpose_linter.py
--- a/scripts/linters/general_purpose_linter.py
+++ b/scripts/linters/general_purpose_linter.py
@@ -25,10 +25,10 @@
 
 import python_utils
 
+from . import js_ts_linter
 from . import linter_utils
 from .. import common
 
-
 EXCLUDED_PATHS = (
     'third_party/*', 'build/*', '.git/*', '*.pyc', 'CHANGELOG',
     'integrations/*', 'integrations_dev/*', '*.svg', '*.gif', '*.png',
@@ -37,7 +37,8 @@
     '*.mp3', '*.mp4', 'node_modules/*', 'typings/*', 'local_compiled_js/*',
     'webpack_bundles/*', 'core/tests/services_sources/*',
     'core/tests/release_sources/tmp_unzip.zip',
-    'core/tests/release_sources/tmp_unzip.tar.gz')
+    'core/tests/release_sources/tmp_unzip.tar.gz',
+    '%s/*' % js_ts_linter.COMPILED_TYPESCRIPT_TMP_PATH)
 
 GENERATED_FILE_PATHS = (
     'extensions/interactions/LogicProof/static/js/generatedDefaultData.ts',
diff --git a/scripts/linters/js_ts_linter.py b/scripts/linters/js_ts_linter.py
--- a/scripts/linters/js_ts_linter.py
+++ b/scripts/linters/js_ts_linter.py
@@ -30,6 +30,7 @@
 import python_utils
 
 from . import linter_utils
+from .. import clean
 from .. import common
 
 CURR_DIR = os.path.abspath(os.getcwd())
@@ -53,6 +54,8 @@
 FILES_EXCLUDED_FROM_ANY_TYPE_CHECK = json.load(python_utils.open_file(
     FILES_EXCLUDED_FROM_ANY_TYPE_CHECK_PATH, 'r'))
 
+COMPILED_TYPESCRIPT_TMP_PATH = 'tmpcompiledjs/'
+
 
 def _get_expression_from_node_if_one_exists(
         parsed_node, components_to_check):
@@ -175,13 +178,11 @@ def _validate_and_parse_js_and_ts_files(self):
                 if filepath.endswith('.js'):
                     raise
                 # Compile typescript file which has syntax invalid for JS file.
-                with linter_utils.temp_dir(prefix='tmpcompiledjs',
-                                           parent=os.getcwd()) as temp_dir:
-                    compiled_js_filepath = self._compile_ts_file(
-                        filepath, temp_dir)
-                    file_content = FILE_CACHE.read(compiled_js_filepath)
-                    parsed_js_and_ts_files[filepath] = esprima.parseScript(
-                        file_content)
+                compiled_js_filepath = self._get_compiled_ts_filepath(filepath)
+
+                file_content = FILE_CACHE.read(compiled_js_filepath)
+                parsed_js_and_ts_files[filepath] = esprima.parseScript(
+                    file_content)
 
         return parsed_js_and_ts_files
 
@@ -210,21 +211,32 @@ def _get_expressions_from_parsed_script(self):
 
         return parsed_expressions_in_files
 
-    def _compile_ts_file(self, filepath, dir_path):
-        """Compiles a typescript file and returns the path for compiled
-        js file.
+    def _get_compiled_ts_filepath(self, filepath):
+        """Returns the path for compiled ts file.
+
+        Args:
+            filepath: str. filepath of ts file
+
+        Returns:
+            str. filepath of compiled ts file.
         """
-        cmd = (
-            './node_modules/typescript/bin/tsc -outDir %s -allowJS %s '
-            '-lib %s -noImplicitUseStrict %s -skipLibCheck '
-            '%s -target %s -typeRoots %s %s typings/*') % (
-                dir_path, 'true', 'es2017,dom', 'true',
-                'true', 'es5', './node_modules/@types', filepath)
-        subprocess.call(cmd, shell=True, stdout=subprocess.PIPE)
         compiled_js_filepath = os.path.join(
-            dir_path, os.path.basename(filepath).replace('.ts', '.js'))
+            os.getcwd(),
+            COMPILED_TYPESCRIPT_TMP_PATH,
+            os.path.relpath(filepath).replace('.ts', '.js'))
         return compiled_js_filepath
 
+    def _compile_all_ts_files(self):
+        """Compiles all project typescript files into
+        COMPILED_TYPESCRIPT_TMP_PATH. Previously, we only compiled
+        the TS files that were needed, but when a relative import was used, the
+        linter would crash with a FileNotFound exception before being able to
+        run. For more details, please see issue #9458.
+        """
+        cmd = ('./node_modules/typescript/bin/tsc -p %s -outDir %s') % (
+            './tsconfig.json', COMPILED_TYPESCRIPT_TMP_PATH)
+        subprocess.call(cmd, shell=True, stdout=subprocess.PIPE)
+
     def _check_any_type(self):
         """Checks if the type of any variable is declared as 'any'
         in TypeScript files.
@@ -756,64 +768,64 @@ def _check_constants_declaration(self):
                     filename_without_extension = filepath[:-3]
                     corresponding_angularjs_filepath = (
                         filename_without_extension + '.ajs.ts')
-                    with linter_utils.temp_dir(parent=os.getcwd()) as temp_dir:
-                        if os.path.isfile(corresponding_angularjs_filepath):
-                            compiled_js_filepath = self._compile_ts_file(
-                                corresponding_angularjs_filepath, temp_dir)
-                            file_content = FILE_CACHE.read(
-                                compiled_js_filepath).decode('utf-8')
-
-                            parsed_script = esprima.parseScript(file_content)
-                            parsed_nodes = parsed_script.body
-                            angularjs_constants_list = []
-                            components_to_check = ['constant']
-                            for parsed_node in parsed_nodes:
-                                expression = (
-                                    _get_expression_from_node_if_one_exists(
-                                        parsed_node, components_to_check))
-                                if not expression:
-                                    continue
-                                else:
-                                    # The following block populates a set to
-                                    # store constants for the Angular-AngularJS
-                                    # constants file consistency check.
-                                    angularjs_constants_name = (
-                                        expression.arguments[0].value)
-                                    angularjs_constants_value = (
-                                        expression.arguments[1].property.name)
-                                    if angularjs_constants_value != (
-                                            angularjs_constants_name):
-                                        failed = True
-                                        python_utils.PRINT(
-                                            '%s --> Please ensure that the '
-                                            'constant %s is initialized '
-                                            'from the value from the '
-                                            'corresponding Angular constants'
-                                            ' file (the *.constants.ts '
-                                            'file). Please create one in the'
-                                            ' Angular constants file if it '
-                                            'does not exist there.' % (
-                                                filepath,
-                                                angularjs_constants_name))
-                                    angularjs_constants_list.append(
-                                        angularjs_constants_name)
-                            angularjs_constants_set = set(
-                                angularjs_constants_list)
-                            if len(angularjs_constants_set) != len(
-                                    angularjs_constants_list):
-                                failed = True
-                                python_utils.PRINT(
-                                    '%s --> Duplicate constant declaration '
-                                    'found.' % (
-                                        corresponding_angularjs_filepath))
-                            angularjs_source_filepaths_to_constants_dict[
-                                corresponding_angularjs_filepath] = (
-                                    angularjs_constants_set)
-                        else:
+
+                    if os.path.isfile(corresponding_angularjs_filepath):
+                        compiled_js_filepath = self._get_compiled_ts_filepath(
+                            corresponding_angularjs_filepath)
+                        file_content = FILE_CACHE.read(
+                            compiled_js_filepath).decode('utf-8')
+
+                        parsed_script = esprima.parseScript(file_content)
+                        parsed_nodes = parsed_script.body
+                        angularjs_constants_list = []
+                        components_to_check = ['constant']
+                        for parsed_node in parsed_nodes:
+                            expression = (
+                                _get_expression_from_node_if_one_exists(
+                                    parsed_node, components_to_check))
+                            if not expression:
+                                continue
+                            else:
+                                # The following block populates a set to
+                                # store constants for the Angular-AngularJS
+                                # constants file consistency check.
+                                angularjs_constants_name = (
+                                    expression.arguments[0].value)
+                                angularjs_constants_value = (
+                                    expression.arguments[1].property.name)
+                                if angularjs_constants_value != (
+                                        angularjs_constants_name):
+                                    failed = True
+                                    python_utils.PRINT(
+                                        '%s --> Please ensure that the '
+                                        'constant %s is initialized '
+                                        'from the value from the '
+                                        'corresponding Angular constants'
+                                        ' file (the *.constants.ts '
+                                        'file). Please create one in the'
+                                        ' Angular constants file if it '
+                                        'does not exist there.' % (
+                                            filepath,
+                                            angularjs_constants_name))
+                                angularjs_constants_list.append(
+                                    angularjs_constants_name)
+                        angularjs_constants_set = set(
+                            angularjs_constants_list)
+                        if len(angularjs_constants_set) != len(
+                                angularjs_constants_list):
                             failed = True
                             python_utils.PRINT(
-                                '%s --> Corresponding AngularJS constants '
-                                'file not found.' % filepath)
+                                '%s --> Duplicate constant declaration '
+                                'found.' % (
+                                    corresponding_angularjs_filepath))
+                        angularjs_source_filepaths_to_constants_dict[
+                            corresponding_angularjs_filepath] = (
+                                angularjs_constants_set)
+                    else:
+                        failed = True
+                        python_utils.PRINT(
+                            '%s --> Corresponding AngularJS constants '
+                            'file not found.' % filepath)
 
                 # Check that the constants are declared only in a
                 # *.constants.ajs.ts file.
@@ -915,6 +927,11 @@ def perform_all_lint_checks(self):
                 'There are no JavaScript or Typescript files to lint.')
             return []
 
+        # Clear temp compiled typescipt files from previous runs.
+        clean.delete_directory_tree(COMPILED_TYPESCRIPT_TMP_PATH)
+        # Compiles all typescipt files into COMPILED_TYPESCRIPT_TMP_PATH.
+        self._compile_all_ts_files()
+
         self.parsed_js_and_ts_files = self._validate_and_parse_js_and_ts_files()
         self.parsed_expressions_in_files = (
             self._get_expressions_from_parsed_script())
@@ -929,6 +946,9 @@ def perform_all_lint_checks(self):
         controller_dependency_messages = (
             self._match_line_breaks_in_controller_dependencies())
 
+        # Clear temp compiled typescipt files.
+        clean.delete_directory_tree(COMPILED_TYPESCRIPT_TMP_PATH)
+
         all_messages = (
             any_type_messages + extra_js_files_messages +
             http_requests_messages + js_and_ts_component_messages +


Test patch: