Repository: mne-tools/mne-python

Issue:
BUG? output when downloading dataset
When downloading a dataset the console output is like this: 
```
[....................................... ] 99.99247 | (589.0 MB / 589.1 MB,   2.5 MB/s)  
[....................................... ] 99.99777 / (589.0 MB / 589.1 MB,   384 kB/s) 
[........................................] 100.00000 - (589.1 MB / 589.1 MB,   2.2 MB/s)
```

isn't the last line supposed to be overwritten by the new line ?  So that only one line is displayed during the whole download


Discussion:
it should indeed....

Are you using an actual pure Python terminal or something else (e.g. one
wrapped by Spyder)? If the latter then it's not our bug but one with their
support of the backspace character.

>

I use an ipython terminal (IPython 6.2.1)
I cannot replicate (IPython 6.3.1, but I doubt this version matters) on Linux (doubt this matters, either). Are you running `ipython` directly from a terminal or is it embedded in some IDE?
Directly from terminal (on Linux fwiw). upgrading to IPython 6.3.1 did not fix
the progress bar is broken in my opinion ... I had similar issues when using the progressbar in autoreject on Travis
Do you get this for:
```
In [1]: print('foo\b\b\bbar')
bar
```
Travis also does not have proper backspace support IIRC (it is a logger so this is expected behavior)
I get only `bar` like you. I have fetched and rebased upstream master to be up-to-date and I still see multiple lines
Actually I don't think we use `\b` but rather `\r`, what does this do:
```
In [1]: print('foo\rbar')
bar
```
And then to really get down to what we do:
```
In [1]: import sys; sys.stdout.write('foo\rbar'); sys.stdout.flush();
bar
```
```
In [17]: print('foo\rbar')
bar
In [18]: import sys; sys.stdout.write('foo\rbar'); sys.stdout.flush();
bar
```

and to push it a bit further: 
```
In [19]: import sys; sys.stdout.write('foo'); sys.stdout.write('\rbar'); sys.stdout.flush();
bar
```

More!
```
In [14]: import mne; pb = mne.utils.ProgressBar(100); pb.update(1); pb.update(50);
[....................                    ] 50.00000 |
```
If this one only prints a single line, then try this one:
```
In [15]: mne.datasets.testing.data_path(force_update=True, verbose=True)
Downloading archive mne-testing-data-0.46.tar.gz to /home/larsoner/mne_data
Downloading https://codeload.github.com/mne-tools/mne-testing-data/tar.gz/0.46 (1 byte)
[........................................] 100.00000 \ ( 80.5 MB /   1 byte,  11.5 MB/s)
```
You should get one updating line (ignore the bad file size, this is a redirect problem).
I get one single line for the first, and an updating line for the second (as should be)

Now I can no longer reproduce : trying to download kiloword or somato, I get the single updated line. I am puzzled.
Ah, starting a new Terminal and a new ipython console gives me multiple lines again:
```
mathurin@mathurin-HP-EliteBook-840-G3:~$ ipython
Python 3.5.4 | packaged by conda-forge | (default, Dec 18 2017, 06:30:33) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import mne

In [2]: mne.datasets.kiloword.data_path()
Using default location ~/mne_data for kiloword...
Downloading archive MNE-kiloword-data.tar.gz to /home/mathurin/mne_data
Downloading https://files.osf.io/v1/resources/rxvq7/providers/osfstorage/59c25bd1594d900250e8a679?action=download&version=1&direct (24.3 MB)
[                                        ] 0.86817 | (  216 kB /  24.3 MB,  17.0
[                                        ] 0.93248 / (  232 kB /  24.3 MB,    85
[                                        ] 0.96464 - (  240 kB /  24.3 MB,    82
```

but (in the same console):
```
In [3]: mne.datasets.testing.data_path(force_update=True, verbose=True)
Downloading archive mne-testing-data-0.46.tar.gz to /home/mathurin/mne_data
Downloading https://codeload.github.com/mne-tools/mne-testing-data/tar.gz/0.46 (900.1 MB)
[                                        ] 0.01823 | (  168 kB / 900.1 MB,   144 kB/s) 
```
has a single line
and then it makes it work for other datasets:
```
In [4]: mne.datasets.kiloword.data_path()
Using default location ~/mne_data for kiloword...
Downloading archive MNE-kiloword-data.tar.gz to /home/mathurin/mne_data
Downloading https://files.osf.io/v1/resources/rxvq7/providers/osfstorage/59c25bd1594d900250e8a679?action=download&version=1&direct (24.3 MB)
[..                                      ] 5.85214 | (  1.4 MB /  24.3 MB,   174 kB/s) 
```


Maybe we need a `flush=True` in the `print` lines or something? I'll think you'll have to debug this locally and try different things to figure it out
It has to do with the length of the line and the size of my terminal window.
Default sized terminal (lines are 80 characters long):
```
In [2]: mne.datasets.kiloword.data_path()
Using default location ~/mne_data for kiloword...
Downloading archive MNE-kiloword-data.tar.gz to /home/mathurin/mne_data
Downloading https://files.osf.io/v1/resources/rxvq7/providers/osfstorage/59c25bd1594d900250e8a679?action=download&version=1&direct (24.3 MB)
[......................                  ] 56.04546 | ( 13.6 MB /  24.3 MB,  18.
[......................                  ] 56.10977 / ( 13.6 MB /  24.3 MB,  25.
[......................                  ] 56.23839 - ( 13.7 MB /  24.3 MB,   33
[......................                  ] 56.36701 \ ( 13.7 MB /  24.3 MB,   32
[......................                  ] 56.49562 | ( 13.7 MB /  24.3 MB,   32
[......................                  ] 56.62424 / ( 13.8 MB /  24.3 MB,   4.
[......................                  ] 56.75286 - ( 13.8 MB /  24.3 MB,   16
[......................                  ] 56.81717 \ ( 13.8 MB /  24.3 MB,   6.
[......................                  ] 56.94579 | ( 13.8 MB /  24.3 MB,   7.
[......................                  ] 57.20303 / ( 13.9 MB /  24.3 MB,  47.
[.......................                 ] 57.71750 - ( 14.0 MB /  24.3 MB,   682 kB/s)   ^C---------------------------------------------------------------------------
KeyboardInterrupt 
```
notice how only the last line has the full speed dispalyed with its unit (682 kB/s). So I believe '\r' is going back to beginning of the buffer, but the first 80 lines of the buffer have already been written ? So it only erases the end of the whole line which is the remaining 10 characters: 
```
In [7]: len('[.......................                 ] 57.71750 - ( 14.0 MB /  24.3 MB,   682 kB
   ...: /s)   ')
Out[7]: 90
```

If I enlarge my terminal window so that the whole line fits then the line is replaced as expected.

Resolving patch:
diff --git a/mne/utils.py b/mne/utils.py
--- a/mne/utils.py
+++ b/mne/utils.py
@@ -1686,15 +1686,22 @@ class ProgressBar(object):
         value, defaults to 0.
     mesg : str
         Message to include at end of progress bar.
-    max_chars : int
-        Number of characters to use for progress bar (be sure to save some room
-        for the message and % complete as well).
+    max_chars : int | str
+        Number of characters to use for progress bar itself.
+        This does not include characters used for the message or percent
+        complete. Can be "auto" (default) to try to set a sane value based
+        on the terminal width.
     progress_character : char
         Character in the progress bar that indicates the portion completed.
     spinner : bool
         Show a spinner.  Useful for long-running processes that may not
         increment the progress bar very often.  This provides the user with
         feedback that the progress has not stalled.
+    max_total_width : int | str
+        Maximum total message width. Can use "auto" (default) to try to set
+        a sane value based on the current terminal width.
+    verbose_bool : bool
+        If True, show progress.
 
     Example
     -------
@@ -1712,11 +1719,11 @@ class ProgressBar(object):
     """
 
     spinner_symbols = ['|', '/', '-', '\\']
-    template = '\r[{0}{1}] {2:.05f} {3} {4}   '
+    template = '\r[{0}{1}] {2:.02f}% {4} {3}   '
 
-    def __init__(self, max_value, initial_value=0, mesg='', max_chars=40,
+    def __init__(self, max_value, initial_value=0, mesg='', max_chars='auto',
                  progress_character='.', spinner=False,
-                 verbose_bool=True):  # noqa: D102
+                 max_total_width='auto', verbose_bool=True):  # noqa: D102
         self.cur_value = initial_value
         if isinstance(max_value, Iterable):
             self.max_value = len(max_value)
@@ -1725,13 +1732,18 @@ def __init__(self, max_value, initial_value=0, mesg='', max_chars=40,
             self.max_value = float(max_value)
             self.iterable = None
         self.mesg = mesg
-        self.max_chars = max_chars
         self.progress_character = progress_character
         self.spinner = spinner
         self.spinner_index = 0
         self.n_spinner = len(self.spinner_symbols)
         self._do_print = verbose_bool
         self.cur_time = time.time()
+        if max_total_width == 'auto':
+            max_total_width = _get_terminal_width()
+        self.max_total_width = int(max_total_width)
+        if max_chars == 'auto':
+            max_chars = min(max(max_total_width - 40, 10), 60)
+        self.max_chars = int(max_chars)
         self.cur_rate = 0
 
     def update(self, cur_value, mesg=None):
@@ -1751,7 +1763,8 @@ def update(self, cur_value, mesg=None):
         cur_time = time.time()
         cur_rate = ((cur_value - self.cur_value) /
                     max(float(cur_time - self.cur_time), 1e-6))
-        # cur_rate += 0.9 * self.cur_rate
+        # Smooth the estimate a bit
+        cur_rate = 0.1 * cur_rate + 0.9 * self.cur_rate
         # Ensure floating-point division so we can get fractions of a percent
         # for the progressbar.
         self.cur_time = cur_time
@@ -1764,9 +1777,8 @@ def update(self, cur_value, mesg=None):
         # Update the message
         if mesg is not None:
             if mesg == 'file_sizes':
-                mesg = '(%s / %s, %s/s)' % (
+                mesg = '(%s, %s/s)' % (
                     sizeof_fmt(self.cur_value).rjust(8),
-                    sizeof_fmt(self.max_value).rjust(8),
                     sizeof_fmt(cur_rate).rjust(8))
             self.mesg = mesg
 
@@ -1778,6 +1790,7 @@ def update(self, cur_value, mesg=None):
                                    progress * 100,
                                    self.spinner_symbols[self.spinner_index],
                                    self.mesg)
+        bar = bar[:self.max_total_width]
         # Force a flush because sometimes when using bash scripts and pipes,
         # the output is not printed until after the program exits.
         if self._do_print:
@@ -1812,6 +1825,14 @@ def __iter__(self):
             self.update_with_increment_value(1)
 
 
+def _get_terminal_width():
+    """Get the terminal width."""
+    if sys.version[0] == '2':
+        return 80
+    else:
+        return shutil.get_terminal_size((80, 20)).columns
+
+
 def _get_ftp(url, temp_file_name, initial_size, file_size, timeout,
              verbose_bool):
     """Safely (resume a) download to a file from FTP."""
@@ -1836,7 +1857,7 @@ def _get_ftp(url, temp_file_name, initial_size, file_size, timeout,
     down_cmd = "RETR " + file_name
     assert file_size == data.size(file_name)
     progress = ProgressBar(file_size, initial_value=initial_size,
-                           max_chars=40, spinner=True, mesg='file_sizes',
+                           spinner=True, mesg='file_sizes',
                            verbose_bool=verbose_bool)
 
     # Callback lambda function that will be passed the downloaded data
@@ -1882,7 +1903,7 @@ def _get_http(url, temp_file_name, initial_size, file_size, timeout,
                            % (total_size, file_size))
     mode = 'ab' if initial_size > 0 else 'wb'
     progress = ProgressBar(total_size, initial_value=initial_size,
-                           max_chars=40, spinner=True, mesg='file_sizes',
+                           spinner=True, mesg='file_sizes',
                            verbose_bool=verbose_bool)
     chunk_size = 8192  # 2 ** 13
     with open(temp_file_name, mode) as local_file:


Test patch: