Repository: google/personfinder

Issue:
Set up Python 3 testing
Most of our tests will fail right now if we try to run them with Python 3, so we'll need to set up something that runs a specific subset of tests; we can add to that set as we make more code Python 3-compatible.


Discussion:
@nworden I am familiar with writing tests. Please let me know if I can help you with this Issue. 
This issue actually isn't about writing the tests themselves, but just having our existing tests run under both versions of Python. It's not very high-priority, because right now hardly any of our tests would pass under Python 3 anyway :(
If you're interested in testing itself, you might prefer an issue like #168 or #490. We also have a couple older issues related to test flakes (like #226).
okay, I will have a look at the linked issue.
Do you have any workflow already designed for migration from `Python 2.7` to  `Python 3`?
We've only just started doing Python 3 work, and we don't expect to actually do the migration anytime soon because App Engine's Python 3 runtime is still in beta. Apart from the library upgrades, I'm starting by running python-modernize over files when I have time to kill, to get them compatible with both Python 2 and 3. TBH, doing that stuff might be particularly difficult for someone who's new to Person Finder, because the parts that python-modernize can't do automatically are parts that might require a good degree of familiarity with Person Finder. I expect I'll have to refer to gimite (who's been working on PF for many years) for help making sure I don't break anything myself, especially around unicode name handling.
okay @nworden. For now, I will try to help in already open issues and get more familiar with the code.

Resolving patch:
diff --git a/app/jautils.py b/app/jautils.py
--- a/app/jautils.py
+++ b/app/jautils.py
@@ -16,12 +16,14 @@
 
 """Utility functions specific for Japanese language."""
 
+from __future__ import absolute_import
+import six
 import re
 import unicodedata
 
 
 # Hiragana to romaji.
-# This table is copied from: http://code.google.com/p/mozc/source/browse/trunk/src/data/preedit/hiragana-romanji.tsv 
+# This table is copied from: http://code.google.com/p/mozc/source/browse/trunk/src/data/preedit/hiragana-romanji.tsv
 HIRAGANA_TO_ROMAJI = [
     [u"う゛ぁ", u"VA", u""],
     [u"う゛ぃ", u"VI", u""],
@@ -473,7 +475,7 @@ def should_normalize(string):
     #  - hiragana
     #  - full/half width katakana
     #  - full width alphabets
-    return re.search(ur'[\u3040-\u30ff\uff00-\uff9f]', string) != None
+    return re.search(six.u(r'[\u3040-\u30ff\uff00-\uff9f]'), string) != None
 
 
 def normalize(string):
@@ -505,7 +507,7 @@ def normalize(string):
 def is_hiragana(string):
     """Returns True if the argument is a non-empty string of only
     hiragana characters."""
-    return re.match(ur'^[\u3040-\u309f]+$', string) != None
+    return re.match(six.u(r'^[\u3040-\u309f]+$'), string) != None
 
 
 def normalize_hiragana(string):
diff --git a/app/text_query.py b/app/text_query.py
--- a/app/text_query.py
+++ b/app/text_query.py
@@ -15,6 +15,7 @@
 
 from __future__ import absolute_import
 import six
+
 __author__ = 'eyalf@google.com (Eyal Fink)'
 
 import unicodedata
@@ -43,7 +44,7 @@ def __init__(self, query):
         # Split out each CJK ideograph as its own word.
         # The main CJK ideograph range is from U+4E00 to U+9FFF.
         # CJK Extension A is from U+3400 to U+4DFF.
-        cjk_separated = re.sub(ur'([\u3400-\u9fff])', r' \1 ', self.normalized)
+        cjk_separated = re.sub(six.u(r'([\u3400-\u9fff])'), r' \1 ', self.normalized)
 
         # Separate the query into words.
         self.words = cjk_separated.split()


Test patch:
diff --git a/tests/test_text_query.py b/tests/test_text_query.py
--- a/tests/test_text_query.py
+++ b/tests/test_text_query.py
@@ -26,7 +26,7 @@ def test_cjk_separation(self):
         q = text_query.TextQuery(u'foo\u4f59\u5609bar\u5e73')
         assert q.words == ['FOO', u'\u4f59', u'\u5609', 'BAR', u'\u5e73']
         assert q.words == q.query_words
-        
+
     def test_parsing(self):
         q = text_query.TextQuery('abcd  e  fghij')
         assert ['ABCD', 'E', 'FGHIJ'] == q.words
diff --git a/tests/unit_tests.py b/tests/unit_tests.py
--- a/tests/unit_tests.py
+++ b/tests/unit_tests.py
@@ -21,15 +21,27 @@
 import argparse
 import os
 import pytest
+import six
 import sys
 
-from google.appengine.api import apiproxy_stub_map
-from google.appengine.api import datastore_file_stub
+# The test files that should be run by default when we're running Python 3.
+PY3_TEST_FILES = [
+    'test_jautils.py',
+    'test_text_query.py',
+]
 
-# Create a new apiproxy and temp datastore to use for this test suite
-apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
-temp_db = datastore_file_stub.DatastoreFileStub('x', None, None, trusted=True)
-apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_db)
+# These dependencies don't support Python 3. Also, they're only used for tests
+# involving App Engine APIs, which we won't be running with Python 3. So, only
+# import them when we're running Python 2.
+if six.PY2:
+    from google.appengine.api import apiproxy_stub_map
+    from google.appengine.api import datastore_file_stub
+
+    # Create a new apiproxy and temp datastore to use for this test suite
+    apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
+    temp_db = datastore_file_stub.DatastoreFileStub(
+        'x', None, None, trusted=True)
+    apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_db)
 
 # An application id is required to access the datastore, so let's create one
 os.environ['APPLICATION_ID'] = 'personfinder-unittest'
@@ -39,6 +51,14 @@
 # When the appserver is running, the APP_DIR should be the current directory...
 os.chdir(os.environ['APP_DIR'])
 
+if six.PY2:
+    default_test_files = [os.environ['TESTS_DIR']]
+else:
+    default_test_files = [
+        os.path.join(os.environ['TESTS_DIR'], test_file)
+        for test_file in PY3_TEST_FILES
+    ]
+
 # ...but we want pytest to default to finding tests in TESTS_DIR, not the cwd.
 # So, when no arguments are given, we use TESTS_DIR by default for the test
 # files we pass to pytest.
@@ -48,7 +68,7 @@
 parser.add_argument('-q', action='store_true')
 parser.add_argument('-k', type=str,
                     help='Keyword expressions to pass to pytest.')
-parser.add_argument('test_files', nargs='*', default=[os.environ['TESTS_DIR']])
+parser.add_argument('test_files', nargs='*', default=default_test_files)
 args = parser.parse_args()
 pytest_args = []
 if args.tb:
diff --git a/tools/all_tests b/tools/all_tests
--- a/tools/all_tests
+++ b/tools/all_tests
@@ -18,6 +18,7 @@
 pushd "$(dirname $0)" >/dev/null && source common.sh && popd >/dev/null
 
 $TOOLS_DIR/unit_tests -q && \
+    $TOOLS_DIR/py3_unit_tests -q && \
     $TOOLS_DIR/server_tests -q && \
     $TOOLS_DIR/python3_compatibility_check && \
     $TOOLS_DIR/ui test && \
diff --git a/tools/py3_unit_tests b/tools/py3_unit_tests
new file mode 100755
--- /dev/null
+++ b/tools/py3_unit_tests
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+# To run all tests that support Python 3 so far:
+#
+#     tools/py3_unit_tests
+#
+# To run just the tests in tests/test_model.py and tests/test_indexing.py:
+#
+#     tools/py3_unit_tests test_model test_indexing
+
+pushd "$(dirname $0)" >/dev/null && source common.sh && popd >/dev/null
+
+# Some tests assumes that the current directory is the root of the working
+# directory.
+cd "$(dirname $0)/.."
+
+echo
+echo "--- Running Python 3 unit tests"
+TZ=UTC $PYTHON3 $TESTS_DIR/unit_tests.py --pyargs --tb=short "$@"
diff --git a/tools/travis_tests b/tools/travis_tests
new file mode 100755
--- /dev/null
+++ b/tools/travis_tests
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Copyright 2010 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runs all the tests for either Python 2.7 or 3.7. Travis will run our tests
+# under two separate environments (one with Python 2.7, one with 3.7), so we
+# can't simply run all the tests at once (like we do locally with
+# tools/all_tests). Instead, we have two separate paths in this file, which will
+# each get executed once on Travis.
+
+pushd "$(dirname $0)" >/dev/null && source common.sh && popd >/dev/null
+
+case "${TRAVIS_PYTHON_VERSION}" in
+
+    2.7)
+        $TOOLS_DIR/unit_tests -q && \
+            $TOOLS_DIR/server_tests -q && \
+            $TOOLS_DIR/ui test && \
+            echo "All Python 2.7 tests passed."
+        ;;
+    3.7)
+        $TOOLS_DIR/py3_unit_tests -q && \
+            $TOOLS_DIR/python3_compatibility_check && \
+            echo "All Python 3.7 tests passed."
+        ;;
+
+ esac