Repository: Qiskit/qiskit

Issue:
Text Drawer fails on windows/appveyor
<!-- ⚠️ If you do not respect this template, your issue will be closed -->
<!-- ⚠️ Make sure to browse the opened and closed issues -->

### Information

- **Qiskit Terra version**: master (but also effects releases)
- **Python version**: Any
- **Operating system**: Windows

### What is the current behavior?

When a test or script prints a circuit using the text drawer as part of CI on windows/appveyor  testing it always raises a UnicodeEncodeError. For example something like:

```
    Traceback (most recent call last):'
     File "C:\\projects\\qiskit-sdk-py\\test\\python\\test_examples.py", line 50, in test_all_examples'
       self.assertEqual(run_example.returncode, 0, error_string)'
     File "C:\\Python37\\lib\\unittest\\case.py", line 839, in assertEqual'
       assertion_func(first, second, msg=msg)'
     File "C:\\Python37\\lib\\unittest\\case.py", line 832, in _baseAssertEqual'
       raise self.failureException(msg)'
    AssertionError: 1 != 0 : Running example circuit_draw.py failed with return code 1'
    'stdout:'
    'stderr: Traceback (most recent call last):\r'
      File "C:\\projects\\qiskit-sdk-py\\examples\\python\\circuit_draw.py", line 34, in <module>\r'
        print(bell_circuit)\r'
     File "C:\\Python37\\lib\\encodings\\cp1252.py", line 19, in encode\r'
       return codecs.charmap_encode(input,self.errors,encoding_table)[0]\r'
    UnicodeEncodeError: 'charmap' codec can't encode characters in position 8-12: character maps to <undefined>\r"
```

Most recently: https://ci.appveyor.com/project/Qiskit/qiskit-sdk-py/builds/25126953/job/9tmvcy2iutrh9qp9#L2063

### Steps to reproduce the problem

Print a circuit to the console during an appveyor job.

### What is the expected behavior?

The circuit prints successfully.

### Suggested solutions

Figure out why this happens in the CI environment and fix it.



Discussion:
Is this still true?
Yes, this is still an issue. I haven't tried unskipping it since we moved to azure pipelines, but it was easy enough to reproduce this failure locally on windows too. It's a real issue, not just CI.
> Yes, this is still an issue. I haven't tried unskipping it since we moved to azure pipelines, but it was easy enough to reproduce this failure locally on windows too. It's a real issue, not just CI.

Is the appveyor code somewhere in the public part of the tree?
@jwoehr Appveyor was just the ci system we used to use for running windows tests (we have since moved away from using it). The issue was running these tests: https://github.com/Qiskit/qiskit-terra/blob/master/test/python/test_examples.py (which just execute the example python scripts) did not work on windows because the character encoding of the cmd shell wasn't utf8 and the text drawer was writing out utf8 characters outside the charmap codec.

This is still a problem with the text drawer, and if you look at my recent comment in #4525 I think the inline version of text drawer output we've added to documentation and release notes is causing failures when trying to read those files on windows (with python's `open`) for the same reason.
> This is still a problem with the text drawer, and if you look at my recent comment in #4525 I think the inline version of text drawer output we've added to documentation and release notes is causing failures when trying to read those files on windows (with python's `open`) for the same reason.

@mtreinish thank you ... I am still interested in #3976 which led to my interest in this problem. That qasm issue has sort of sat in jail for some months ... wondering if I should resume work and remove the tests for now so we can get on with it? @1ucian0 ?


Resolving patch:
diff --git a/qiskit/visualization/circuit_visualization.py b/qiskit/visualization/circuit_visualization.py
--- a/qiskit/visualization/circuit_visualization.py
+++ b/qiskit/visualization/circuit_visualization.py
@@ -405,7 +405,7 @@ def qx_color_scheme():
 def _text_circuit_drawer(circuit, filename=None, reverse_bits=False,
                          plot_barriers=True, justify=None, vertical_compression='high',
                          idle_wires=True, with_layout=True, fold=None, initial_state=True,
-                         cregbundle=False):
+                         cregbundle=False, encoding=None):
     """Draws a circuit using ascii art.
 
     Args:
@@ -426,8 +426,9 @@ def _text_circuit_drawer(circuit, filename=None, reverse_bits=False,
                     `shutil.get_terminal_size()`. If you don't want pagination
                    at all, set `fold=-1`.
         initial_state (bool): Optional. Adds |0> in the beginning of the line. Default: `True`.
-        cregbundle (bool): Optional. If set True bundle classical registers. Only used by
-            the ``text`` output. Default: ``False``.
+        cregbundle (bool): Optional. If set True bundle classical registers. Default: ``False``.
+        encoding (str): Optional. Sets the encoding preference of the output.
+                   Default: ``sys.stdout.encoding``.
     Returns:
         TextDrawing: An instances that, when printed, draws the circuit in ascii art.
     """
@@ -441,13 +442,14 @@ def _text_circuit_drawer(circuit, filename=None, reverse_bits=False,
         layout = None
     global_phase = circuit.global_phase if hasattr(circuit, 'global_phase') else None
     text_drawing = _text.TextDrawing(qregs, cregs, ops, layout=layout, initial_state=initial_state,
-                                     cregbundle=cregbundle, global_phase=global_phase)
+                                     cregbundle=cregbundle, global_phase=global_phase,
+                                     encoding=encoding)
     text_drawing.plotbarriers = plot_barriers
     text_drawing.line_length = fold
     text_drawing.vertical_compression = vertical_compression
 
     if filename:
-        text_drawing.dump(filename)
+        text_drawing.dump(filename, encoding=encoding)
     return text_drawing
 
 
diff --git a/qiskit/visualization/text.py b/qiskit/visualization/text.py
--- a/qiskit/visualization/text.py
+++ b/qiskit/visualization/text.py
@@ -36,6 +36,11 @@ class TextDrawerCregBundle(VisualizationError):
     pass
 
 
+class TextDrawerEncodingError(VisualizationError):
+    """A problem with encoding"""
+    pass
+
+
 class DrawElement():
     """ An element is an instruction or an operation that need to be drawn."""
 
@@ -517,13 +522,12 @@ class TextDrawing():
 
     def __init__(self, qregs, cregs, instructions, plotbarriers=True,
                  line_length=None, vertical_compression='high', layout=None, initial_state=True,
-                 cregbundle=False, global_phase=None):
+                 cregbundle=False, global_phase=None, encoding=None):
         self.qregs = qregs
         self.cregs = cregs
         self.instructions = instructions
         self.layout = layout
         self.initial_state = initial_state
-
         self.cregbundle = cregbundle
         self.global_phase = global_phase
         self.plotbarriers = plotbarriers
@@ -531,6 +535,7 @@ def __init__(self, qregs, cregs, instructions, plotbarriers=True,
         if vertical_compression not in ['high', 'medium', 'low']:
             raise ValueError("Vertical compression can only be 'high', 'medium', or 'low'")
         self.vertical_compression = vertical_compression
+        self.encoding = encoding if encoding else sys.stdout.encoding
 
     def __str__(self):
         return self.single_string()
@@ -548,20 +553,25 @@ def __repr__(self):
 
     def single_string(self):
         """Creates a long string with the ascii art.
-
         Returns:
             str: The lines joined by a newline (``\\n``)
         """
-        return "\n".join(self.lines())
-
-    def dump(self, filename, encoding="utf8"):
+        try:
+            return "\n".join(self.lines()).encode(self.encoding).decode(self.encoding)
+        except (UnicodeEncodeError, UnicodeDecodeError):
+            warn('The encoding %s has a limited charset. Consider a different encoding in your '
+                 'environment. UTF-8 is being used instead' % self.encoding, RuntimeWarning)
+            self.encoding = 'utf-8'
+            return "\n".join(self.lines()).encode(self.encoding).decode(self.encoding)
+
+    def dump(self, filename, encoding=None):
         """Dumps the ascii art in the file.
 
         Args:
             filename (str): File to dump the ascii art.
-            encoding (str): Optional. Default "utf-8".
+            encoding (str): Optional. Force encoding, instead of self.encoding.
         """
-        with open(filename, mode='w', encoding=encoding) as text_file:
+        with open(filename, mode='w', encoding=encoding or self.encoding) as text_file:
             text_file.write(self.single_string())
 
     def lines(self, line_length=None):


Test patch:
diff --git a/test/python/test_examples.py b/test/python/test_examples.py
--- a/test/python/test_examples.py
+++ b/test/python/test_examples.py
@@ -30,7 +30,6 @@
 class TestPythonExamples(QiskitTestCase):
     """Test example scripts"""
 
-    @unittest.skipIf(os.name == 'nt', 'Skip on windows until #2616 is fixed')
     @unittest.skipIf(sys.platform == 'darwin' and sys.version_info[1] == 8,
                      "Multiprocess spawn fails on macOS python 3.8 without "
                      "__name__ == '__main__' guard")
@@ -44,15 +43,15 @@ def test_all_examples(self):
                 example_path = os.path.join(examples_dir, example)
                 cmd = [sys.executable, example_path]
                 run_example = subprocess.Popen(cmd, stdout=subprocess.PIPE,
-                                               stderr=subprocess.PIPE)
+                                               stderr=subprocess.PIPE,
+                                               env={**os.environ, "PYTHONIOENCODING": "utf8"})
                 stdout, stderr = run_example.communicate()
                 error_string = "Running example %s failed with return code %s\n" % (
                     example, run_example.returncode)
                 error_string += "stdout:%s\nstderr: %s" % (
-                    stdout.decode('utf8'), stderr.decode('utf8'))
+                    stdout, stderr)
                 self.assertEqual(run_example.returncode, 0, error_string)
 
-    @unittest.skipIf(os.name == 'nt', 'Skip on windows until #2616 is fixed')
     @unittest.skipIf(sys.platform == 'darwin' and sys.version_info[1] == 8,
                      "Multiprocess spawn fails on macOS python 3.8 without "
                      "__name__ == '__main__' guard")
@@ -72,7 +71,8 @@ def test_all_ibmq_examples(self, qe_token, qe_url):
                 example_path = os.path.join(ibmq_examples_dir, example)
                 cmd = [sys.executable, example_path]
                 run_example = subprocess.Popen(cmd, stdout=subprocess.PIPE,
-                                               stderr=subprocess.PIPE)
+                                               stderr=subprocess.PIPE,
+                                               env={**os.environ, "PYTHONIOENCODING": "utf8"})
                 stdout, stderr = run_example.communicate()
                 error_string = "Running example %s failed with return code %s\n" % (
                     example, run_example.returncode)
diff --git a/test/python/visualization/references/circuit_text_ref_cp437.txt b/test/python/visualization/references/circuit_text_ref_cp437.txt
new file mode 100644
--- /dev/null
+++ b/test/python/visualization/references/circuit_text_ref_cp437.txt
@@ -0,0 +1,13 @@
+        ���Ŀ���Ŀ���Ŀ � ���Ŀ���Ŀ�����Ŀ���Ŀ�����Ŀ����Ŀ������Ŀ���Ŀ     ��������Ŀ��������Ŀ��������Ŀ��������Ŀ�����������Ŀ��������������Ŀ                                                                                �Ŀ      
+q_0: |0>� X ô Y ô Z �İĴ H ô S ô SDG ô T ô TDG ô �X ô �XDG ô I ��|0>Ĵ RX(pi) ô RY(pi) ô RZ(pi) ô U1(pi) ô U2(pi,pi) ô U3(pi,pi,pi) ��X�����������������������������������������������������������������������������ĴM�������
+        �İ������������ � ������������������������������������������������     ��������������������������������������������������������������������� � ���Ŀ���Ŀ � ���Ŀ �pi ��������������Ŀ��������Ŀ��������Ŀ��������Ŀ  �   � ����Ŀ   
+q_1: |0>�İ������������������������������������������������������������������������������������������������������������������������������������������XĴ X ô Y ���Ĵ H �����Ĵ U3(pi,pi,pi) ô RZ(pi) ô RY(pi) ô RX(pi) �������X���ĴM����
+          �                                                                                                                                            ����������   �����     �������������������������������������������������Ŀ �  � ����Ŀ
+q_2: |0>�İ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Ĵ X ��X������ĴM�
+          �                                                                                                                                                                                                                 �����    �  � ���
+ c_0: 0 �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
+                                                                                                                                                                                                                                        �  � 
+ c_1: 0 �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
+                                                                                                                                                                                                                                           � 
+ c_2: 0 �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
+                                                                                                                                                                                                                                             
\ No newline at end of file
diff --git a/test/python/visualization/references/circuit_text_ref_original.txt b/test/python/visualization/references/circuit_text_ref_original.txt
deleted file mode 100644
--- a/test/python/visualization/references/circuit_text_ref_original.txt
+++ /dev/null
@@ -1,12 +0,0 @@
-        ┌───┐┌───┐┌───┐ ░ ┌───┐┌───┐┌─────┐┌───┐┌─────┐┌────┐     ┌────────┐┌────────┐┌────────┐┌────────┐┌───────────┐┌──────────────┐                                                                                ┌─┐
-q_0: |0>┤ X ├┤ Y ├┤ Z ├─░─┤ H ├┤ S ├┤ Sdg ├┤ T ├┤ Tdg ├┤ Id ├─|0>─┤ Rx(pi) ├┤ Ry(pi) ├┤ Rz(pi) ├┤ U1(pi) ├┤ U2(pi,pi) ├┤ U3(pi,pi,pi) ├─X───■────■───■───■───■──────────■────────────■─────────■─────────■───────■───■─┤M├──────
-        └─░─┘└───┘└───┘ ░ └───┘└───┘└─────┘└───┘└─────┘└────┘     └────────┘└────────┘└────────┘└────────┘└───────────┘└──────────────┘ │ ┌─┴─┐┌─┴─┐ │ ┌─┴─┐ │pi ┌──────┴───────┐┌───┴────┐┌───┴────┐┌───┴────┐  │   │ └╥┘┌─┐   
-q_1: |0>──░─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────X─┤ X ├┤ Y ├─■─┤ H ├─■───┤ U3(pi,pi,pi) ├┤ Rz(pi) ├┤ Ry(pi) ├┤ Rx(pi) ├──■───X──╫─┤M├───
-          ░                                                                                                                               └───┘└───┘   └───┘     └──────────────┘└────────┘└────────┘└────────┘┌─┴─┐ │  ║ └╥┘┌─┐
-q_2: |0>──░────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ X ├─X──╫──╫─┤M├
-          ░                                                                                                                                                                                                    └───┘    ║  ║ └╥┘
- c_0: 0 ════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╩══╬══╬═
-                                                                                                                                                                                                                           ║  ║
- c_1: 0 ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╩══╬═
-                                                                                                                                                                                                                              ║
- c_2: 0 ══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╩═
diff --git a/test/python/visualization/references/circuit_text_ref.txt b/test/python/visualization/references/circuit_text_ref_utf8.txt
similarity index 100%
rename from test/python/visualization/references/circuit_text_ref.txt
rename to test/python/visualization/references/circuit_text_ref_utf8.txt
diff --git a/test/python/visualization/test_circuit_visualization_output.py b/test/python/visualization/test_circuit_visualization_output.py
--- a/test/python/visualization/test_circuit_visualization_output.py
+++ b/test/python/visualization/test_circuit_visualization_output.py
@@ -21,6 +21,7 @@
 from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister
 
 from qiskit.tools.visualization import HAS_MATPLOTLIB, circuit_drawer
+from qiskit.visualization.circuit_visualization import _text_circuit_drawer
 
 from .visualization import QiskitVisualizationTestCase, path_to_diagram_reference
 
@@ -30,7 +31,8 @@ class TestCircuitVisualizationImplementation(QiskitVisualizationTestCase):
 
     latex_reference = path_to_diagram_reference('circuit_latex_ref.png')
     matplotlib_reference = path_to_diagram_reference('circuit_matplotlib_ref.png')
-    text_reference = path_to_diagram_reference('circuit_text_ref.txt')
+    text_reference_utf8 = path_to_diagram_reference('circuit_text_ref_utf8.txt')
+    text_reference_cp437 = path_to_diagram_reference('circuit_text_ref_cp437.txt')
 
     def sample_circuit(self):
         """Generate a sample circuit that includes the most common elements of
@@ -97,16 +99,28 @@ def test_matplotlib_drawer(self):
         self.assertImagesAreEqual(filename, self.matplotlib_reference)
         os.remove(filename)
 
-    def test_text_drawer(self):
-        filename = self._get_resource_path('current_textplot.txt')
+    def test_text_drawer_utf8(self):
+        filename = self._get_resource_path('current_textplot_utf8.txt')
         qc = self.sample_circuit()
-        output = circuit_drawer(qc, filename=filename, output="text", fold=-1, initial_state=True,
-                                cregbundle=False)
+        output = _text_circuit_drawer(qc, filename=filename, fold=-1,
+                                      initial_state=True, cregbundle=False, encoding='utf8')
+        try:
+            encode(str(output), encoding='utf8')
+        except UnicodeEncodeError:
+            self.fail("_text_circuit_drawer() should be utf8.")
+        self.assertFilesAreEqual(filename, self.text_reference_utf8, 'utf8')
+        os.remove(filename)
+
+    def test_text_drawer_cp437(self):
+        filename = self._get_resource_path('current_textplot_cp437.txt')
+        qc = self.sample_circuit()
+        output = _text_circuit_drawer(qc, filename=filename, fold=-1,
+                                      initial_state=True, cregbundle=False, encoding='cp437')
         try:
             encode(str(output), encoding='cp437')
         except UnicodeEncodeError:
-            self.fail("_text_circuit_drawer() should only use extended ascii (aka code page 437).")
-        self.assertFilesAreEqual(filename, self.text_reference)
+            self.fail("_text_circuit_drawer() should be cp437.")
+        self.assertFilesAreEqual(filename, self.text_reference_cp437, 'cp437')
         os.remove(filename)
 
 
diff --git a/test/python/visualization/visualization.py b/test/python/visualization/visualization.py
--- a/test/python/visualization/visualization.py
+++ b/test/python/visualization/visualization.py
@@ -35,12 +35,12 @@ def _this_directory():
 class QiskitVisualizationTestCase(QiskitTestCase):
     """Visual accuracy of visualization tools outputs tests."""
 
-    def assertFilesAreEqual(self, current, expected):
+    def assertFilesAreEqual(self, current, expected, encoding='cp437'):
         """Checks if both file are the same."""
         self.assertTrue(os.path.exists(current))
         self.assertTrue(os.path.exists(expected))
-        with open(current, encoding='cp437') as cur, \
-                open(expected, encoding='cp437') as exp:
+        with open(current, encoding=encoding) as cur, \
+                open(expected, encoding=encoding) as exp:
             self.assertEqual(cur.read(), exp.read())
 
     def assertImagesAreEqual(self, current, expected, diff_tolerance=0.001):