Repository: ipython/ipython

Issue:
investigate os.scandir
Starting with Python 3.5 os.scandir is available and is faster than listdir. It is also availlable on PyPI there : https://pypi.python.org/pypi/scandir

Looking as the various usage of listdir and os.walk on our codebase and seeing whether we can replace their usage by scandir/scandir walk should be a relatively simple task that might lead to some performance improvements and maybe even code simplification. 

We should also investigate this on jupyter/notebook and jupyter/nbconvert likely (cc @michaelpacer) 



Discussion:
To clarify a bit, it's not that `scandir` is faster than `listdir` by itself. It gets extra information compared to listdir, so it should be faster than using `listdir` _plus_ calling something like `os.path.isdir(p)` on each file it returns. We do do precisely that in some places, though.

I believe that `os.walk` uses scandir internally, so it should get a free speed boost without us doing anything.

> To clarify a bit, it's not that scandir is faster than listdir by itself. It gets extra information compared to listdir, so it should be faster than using listdir plus calling something like os.path.isdir(p) on each file it returns. We do do precisely that in some places, though.
> 
> I believe that os.walk uses scandir internally, so it should get a free speed boost without us doing anything.

That's only on 3.5+ IIRC, we can also depend on the scandir package for 3.3 and 3.4 then use the listdir and walk provided by these packages.

Eh, I doubt it's worth adding a dependency for a probably trivial speed-up.

I will like to work on this one, do you think guys we should go on for it?
@Carreau ? @takluyver ?
It would be useful if you could find any places in IPython's code which might benefit from `scandir()` - typically these are places where `listdir()` is used to find files in a directory, followed by checking `isdir()` or `isfile()` on each one. If there aren't any such cases in IPython, we can close this.

Scandir is new in Python 3.5, and IPython still supports Python 3.4 for now, so we won't make the changes until we drop support for that version. But knowing where to change at that point would be useful.
Ok, I will do this, thanks:p
Not sure if anyone is still working on this. I might take a look
@takluyver said 
> Scandir is new in Python 3.5, and IPython still supports Python 3.4 for now, so we won't make the changes until we drop support for that version. But knowing where to change at that point would be useful.

Do you still want to just mark the locations to make these changes? Or did we want to look into a module that supports both ways until 3.4 is dropped. I think 3.4 is still supported.

> I think 3.4 is still supported.

3.4 has been dropped, at least in IPython.
Already marking the area where there would be a potential speedup would be a useful first step, in IPython but also in notebook, or any other package.

It may be that no code would actually be better and we can close the issue. Which is already a step forward. 
@Carreau So are you saying this ticket finished? If so I can look for something else...
> So are you saying this ticket finished

No, far from it. I'm saying we don't know if something need to be done or not. 
FOr example [this](https://github.com/ipython/ipython/blob/173717906c8643a6d5b6fb2ec33e8e96dacc63f0/IPython/core/profileapp.py#L97-L108) likely can use scandir, beasu we use `os.path.isdir(full_path)`, but one need to look to each usage carefully. 
got it. Sorry for confusion. I will get on this asap.

Resolving patch:
diff --git a/IPython/core/magics/osm.py b/IPython/core/magics/osm.py
--- a/IPython/core/magics/osm.py
+++ b/IPython/core/magics/osm.py
@@ -31,6 +31,51 @@ class OSMagics(Magics):
     """Magics to interact with the underlying OS (shell-type functionality).
     """
 
+    def __init__(self, shell=None, **kwargs):
+
+        # Now define isexec in a cross platform manner.
+        self.is_posix = False
+        self.execre = None
+        if os.name == 'posix':
+            self.is_posix = True
+        else:
+            try:
+                winext = os.environ['pathext'].replace(';','|').replace('.','')
+            except KeyError:
+                winext = 'exe|com|bat|py'
+            
+            self.execre = re.compile(r'(.*)\.(%s)$' % winext,re.IGNORECASE)
+
+        # call up the chain
+        super().__init__(shell=shell, **kwargs)
+
+
+    @skip_doctest
+    def _isexec_POSIX(self, file):
+        """
+            Test for executible on a POSIX system
+        """
+        return file.is_file() and os.access(file.path, os.X_OK)
+
+    
+    @skip_doctest
+    def _isexec_WIN(self, file):
+        """
+            Test for executible file on non POSIX system
+        """
+        return file.is_file() and self.execre.match(file.name) is not None
+
+    @skip_doctest
+    def isexec(self, file):
+        """
+            Test for executible file on non POSIX system
+        """
+        if self.is_posix:
+            return self._isexec_POSIX(file)
+        else:
+            return self._isexec_WIN(file)
+
+
     @skip_doctest
     @line_magic
     def alias(self, parameter_s=''):
@@ -160,65 +205,59 @@ def rehashx(self, parameter_s=''):
             os.environ.get('PATH','').split(os.pathsep)]
 
         syscmdlist = []
-        # Now define isexec in a cross platform manner.
-        if os.name == 'posix':
-            isexec = lambda fname:os.path.isfile(fname) and \
-                     os.access(fname,os.X_OK)
-        else:
-            try:
-                winext = os.environ['pathext'].replace(';','|').replace('.','')
-            except KeyError:
-                winext = 'exe|com|bat|py'
-            if 'py' not in winext:
-                winext += '|py'
-            execre = re.compile(r'(.*)\.(%s)$' % winext,re.IGNORECASE)
-            isexec = lambda fname:os.path.isfile(fname) and execre.match(fname)
         savedir = os.getcwd()
 
         # Now walk the paths looking for executables to alias.
         try:
             # write the whole loop for posix/Windows so we don't have an if in
             # the innermost part
-            if os.name == 'posix':
+            if self.is_posix:
                 for pdir in path:
                     try:
                         os.chdir(pdir)
-                        dirlist = os.listdir(pdir)
                     except OSError:
                         continue
+
+                    # for python 3.6+ rewrite to: with os.scandir(pdir) as dirlist:
+                    dirlist = os.scandir(path=pdir)
                     for ff in dirlist:
-                        if isexec(ff):
+                        if self.isexec(ff):
+                            fname = ff.name
                             try:
                                 # Removes dots from the name since ipython
                                 # will assume names with dots to be python.
-                                if not self.shell.alias_manager.is_alias(ff):
+                                if not self.shell.alias_manager.is_alias(fname):
                                     self.shell.alias_manager.define_alias(
-                                        ff.replace('.',''), ff)
+                                        fname.replace('.',''), fname)
                             except InvalidAliasError:
                                 pass
                             else:
-                                syscmdlist.append(ff)
+                                syscmdlist.append(fname)
             else:
                 no_alias = Alias.blacklist
                 for pdir in path:
                     try:
                         os.chdir(pdir)
-                        dirlist = os.listdir(pdir)
                     except OSError:
                         continue
+
+                    # for python 3.6+ rewrite to: with os.scandir(pdir) as dirlist:
+                    dirlist = os.scandir(pdir)
                     for ff in dirlist:
-                        base, ext = os.path.splitext(ff)
-                        if isexec(ff) and base.lower() not in no_alias:
+                        fname = ff.name
+                        base, ext = os.path.splitext(fname)
+                        if self.isexec(ff) and base.lower() not in no_alias:
                             if ext.lower() == '.exe':
-                                ff = base
+                                fname = base
                                 try:
                                     # Removes dots from the name since ipython
                                     # will assume names with dots to be python.
                                     self.shell.alias_manager.define_alias(
-                                        base.lower().replace('.',''), ff)
+                                        base.lower().replace('.',''), fname)
                                 except InvalidAliasError:
                                     pass
-                                syscmdlist.append(ff)
+                                syscmdlist.append(fname)
+
             self.shell.db['syscmdlist'] = syscmdlist
         finally:
             os.chdir(savedir)
diff --git a/IPython/core/profileapp.py b/IPython/core/profileapp.py
--- a/IPython/core/profileapp.py
+++ b/IPython/core/profileapp.py
@@ -96,27 +96,26 @@
 
 def list_profiles_in(path):
     """list profiles in a given root directory"""
-    files = os.listdir(path)
     profiles = []
+
+    # for python 3.6+ rewrite to: with os.scandir(path) as dirlist:
+    files = os.scandir(path)
     for f in files:
-        try:
-            full_path = os.path.join(path, f)
-        except UnicodeError:
-            continue
-        if os.path.isdir(full_path) and f.startswith('profile_'):
-            profiles.append(f.split('_',1)[-1])
+        if f.is_dir() and f.name.startswith('profile_'):
+            profiles.append(f.name.split('_', 1)[-1])
     return profiles
 
 
 def list_bundled_profiles():
     """list profiles that are bundled with IPython."""
     path = os.path.join(get_ipython_package_dir(), u'core', u'profile')
-    files = os.listdir(path)
     profiles = []
+
+    # for python 3.6+ rewrite to: with os.scandir(path) as dirlist:
+    files =  os.scandir(path)
     for profile in files:
-        full_path = os.path.join(path, profile)
-        if os.path.isdir(full_path) and profile != "__pycache__":
-            profiles.append(profile)
+        if profile.is_dir() and profile.name != "__pycache__":
+            profiles.append(profile.name)
     return profiles
 
 


Test patch: