Repository: browniebroke/django-codemod

Issue:
codemod goes all over the place and triggers OOM
I was building my first fixer, found at its current horrid state at https://github.com/browniebroke/django-codemod/pull/123 (4803651c26), then I ran the following on a file which definitely needs the update

```sh
djcodemod --removed-in 2.0 redirect_urls/utils.py
```
The file in question can be found at https://github.com/pmac/django-redirect-urls/

I go get a coffee, and get otherwise distracted, etc, etc.  A few hours later I come back and find my machine is out of memory, and codemod is having fun in my [pnpm](https://github.com/pnpm/pnpm) global node_modules , which is rather large.

I am sure I will find out how and why, but it isnt immediately clear to me, and I dare say something needs to prevent this happening to other people.

I had thousands of these:

```py
Codemodding /home/jayvdb/.asdf/installs/nodejs/8.14.0/.npm/pnpm-global/1/node_modules/.registry.npmjs.org/node-gyp/3.8.0/node_modules/node-gyp/gyp/pylib/gyp/xcode_ninja.py
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(,)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 283, in _parallel_exec_process_stub
    input_tree = parse_module(
  File "/usr/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 71, in parse_module
    result = _parse(
  File "/usr/lib/python3.8/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
    result = parser.parse()
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 111, in parse
    self._add_token(token)
  File "/usr/lib/python3.8/site-packages/libcst/_parser/base_parser.py", line 187, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 31:17.
Incomplete input. Encountered ',', but expected ':'.

  except OSError, e:
                ^

Failed to codemod /home/jayvdb/.asdf/installs/nodejs/8.14.0/.npm/pnpm-global/1/node_modules/.registry.npmjs.org/node-gyp/3.8.0/node_modules/node-gyp/gyp/pylib/gyp/xcode_ninja.py
```

Here is the output at the end when it was killed (kinda nice that my oom killer did kill this and not my desktop apps, which it often targets first)

```
38m 42s 0.0540% complete, 1194h 25m 20s estimated for 1362939 files to go...Terminated
```


Discussion:
Hum, that's strange indeed. It seems to be hanging for a while without any output. 

I've a made some testing, and it seems to work fine if the path is `.` or a single file in the current directory, for example `utils.py`, but the issue appears when the path has a `/` in it.
Ok found the problem, it's the way we call `libCST`'s `gather_files`: it expects a sequence of paths, but I pass only a single one. So it aplit by letters and process each 'path' in the sequence. Each path being a one letter path.

Resolving patch:
diff --git a/django_codemod/cli.py b/django_codemod/cli.py
--- a/django_codemod/cli.py
+++ b/django_codemod/cli.py
@@ -136,7 +136,7 @@ class CustomCommand(BaseCodemodCommand):
 
 def call_command(command_instance: BaseCodemodCommand, path: str):
     """Call libCST with our customized command."""
-    files = gather_files(path)
+    files = gather_files([path])
     try:
         # Super simplified call
         result = parallel_exec_transform_with_prettyprint(


Test patch: