Package: lib3to6
lib3to6Compile Python 3.6+ code to Python 2.7+ compatible code. The idea is
quite similar to Babelhttps://babeljs.io/. Develop using the newest
interpreter and use (most) new language features and still maintain
backward compatibility.Project/Repo:Code Quality/CI:NamerolesinceuntilManuel Barkhau (mbarkhau@gmail.com)author/maintainer2018-09-Ease the Transition from Old InterpretersPython Versions and CompatibilityUsage CaveatsPer-File Opt-In/Opt-OutIntegration usingsetup.pyAutomatic ConversionsMotivationHow it worksContributingProject Status (as of 2020-09-01): BetaFuture WorkAlternativesFAQMotivation: Ease the Transition from Old InterpretersEspecially for packages, you probably don't want to force all your
users to already use python 3.10, so maintaining backward
compatibility is a good idea. Nonetheless, for development it is a
good idea to use the newest interpreter you can and ideally you can
use the newer type annotation features to type-check your code, but
ellide these annotations in the distribution, since they are not
needed at runtime.If your existing project uses Python2.7, it may not be possible to
dedicate a substantial block of time to update all of your code and
then flip a switch to start running on Python3. To make matters worse,
you may continue to write code only for Python2.7, since that is what
your production code will actually run on. Withlib3to6you can
start to use Python3 for development and integration (ensuring forward
compatibility) and still maintain backward compatibility while you
have to deploy for Python2.Python Versions and CompatibilityThe compiled output is tested using:Python 3.9Python 3.8Python 3.7Python 3.6Python 3.5Python 2.7PyPy 3.6PyPy 3.5The test-suite for the transpiler is run using:Python 3.9Python 3.8Python 3.7Python 3.6PyPy 3.6The compiled output may work with other versions of python, such as<=2.6or>=3.0 <=3.4, but these are not tested.Usage Caveatslib3to6does not add any runtime dependencies of its own, but it does inject code, such as temporary variables and imports from the standard library (itertoolsandbuiltinsin particular). Any changes will only add a constantO(1)overhead.lib3to6does optimistic ast transformations, with the assumption that you're not doing anything too crazy in your code. An example of such a transformation is the support forPEP3102 - Keyword-Only Arguments.lib3to6will change the function signature to use**kwargsand add locals extracted fromkwargs.$ cat kwonly_args_demo.py
def compare(a, b, *, key=None):
    ...$ lib3to6 kwonly_args_demo.py
def compare(a, b, **kwargs):
    key = kwargs.get('key', None)
    passThis means that the function signature you can get using theinspectmodule may not be what you expect for the output oflib3to6.Per-File Opt-In/Opt-OutSincelib3to6==v202008.1042there is support to selectively enable/disable transpilation on a per-file basis.Any file which starts with a# lib3to6: disabledcomment, will not be transpiled. For these, you will have to take care of forward/backward compatibility yourself.# -*- coding: utf-8 -*-# lib3to6: disabled"""A module written to work both with Python2 and 3.This module doesn't need to be transpiled by lib3to6."""from__future__importprint_function...importsysPY3=sys.version_info[0]>2ifPY3:...else:...Instead of opt-out, you can also take an opt-in approach. You will have to switch thedefault_modeargument:# setup.pypackage_dir={"":"src"}ifany(arg.startswith("bdist")forarginsys.argv):importlib3to6package_dir=lib3to6.fix(package_dir,default_mode='disabled')This will leave all files untouched, except for those marked with a# lib3to6: enabledcomment.# lib3to6: enabled"""A module written to work both with Python2 and 3.This module doesn't need to be transpiled by lib3to6."""name:str="Wörld"print(f"Hello{world}!")Integration usingsetup.pyThe cli commandlib3to6 <filename>is nice for demo purposes, but for integration with your project, you may prefer to use it in yoursetup.pyfile. Contributions for other kinds of integration are most welcome.# setup.pyimportsysimportsetuptoolspackages=setuptools.find_packages(".")package_dir={"":"."}install_requires=['typing;python_version<"3.5"']ifany(arg.startswith("bdist")forarginsys.argv):importlib3to6package_dir=lib3to6.fix(package_dir,target_version="2.7",install_requires=install_requires,default_mode='enabled',)setuptools.setup(name="my-module",version="0.1.0",packages=packages,package_dir=package_dir,install_requires=install_requires,classifiers=["Programming Language :: Python","Programming Language :: Python :: 2","Programming Language :: Python :: 3",...],)When you build you package, the contents of the resulting distribution
will be the code that was converted by lib3to6.~/my-module$pythonsetup.pybdist_wheel--python-tag=py2.py3
runningbdist_wheel
...
~/my-module$ls-1dist/
my_module-201808.1-py2.py3-none-any.whl

~/my-module$python3-mpipinstalldist/my_module-201808.1-py2.py3-none-any.whl
Processing./dist/my_module-201808.1-py2.py3-none-any.whl
Installingcollectedpackages:my-module
Successfullyinstalledmy-module-201808.1

~/my-module$python2-mpipinstalldist/my_module-201808.1-py2.py3-none-any.whl
Processing./dist/my_module-201808.1-py2.py3-none-any.whl
Installingcollectedpackages:my-module
Successfullyinstalledmy-module-201808.1When testing, make sure you're not importingmy_modulefrom your local
directory, which is probably the original source code. Instead you can
either manipulate yourPYTHONPATH, or simply switch directories...~/$python3-c"import my_module"/home/user/my-module/my_module/__init__.py
Hello世界from3.6.5!

~/my-module$cd..
~/$python3-c"import my_module"/home/user/envs/py36/lib/python3.6/site-packages/my_module/__init__.py
Hello世界from3.6.5!

~$python2-c"import my_module"/home/user/envs/py27/lib/python2.7/site-packages/my_module/__init__.py
Hello世界from2.7.15!Automatic ConversionsNot all new language features have a semantic equivalent in older
versions. To the extent these can be detected, an error will be
reported when these features are used.Note that a fix is not applied if the lowest version of python that
you are targeting already supports the newer syntax. The conversions
are ordered by when the feature was introduced.PEP 572: Assignment Expressions (aka. the walrus operator)# Since 3.8ifmatch1:=pattern1.match(data):result=match1.group(1)# From 2.7 to 3.7match1=pattern1.match(data)ifmatch1:result=match1.group(1)Some expressions nested expressions in a condition are not so easy,
in which case lib3to6 will bend over backwards.# Since 3.8while(block:=f.read(4096))!='':process(block)# For [2.7 - 3.7]__loop_condition=Truewhile__loop_condition:block=f.read(4096)__loop_condition=block!=''if__loop_condition:process(block)PEP 563: Postponed Evaluation of Annotations# Since 3.7classSelfRef:defmethod(self)->SelfRef:pass# From 3.0 to 3.6classSelfRef:defmethod(self)->'SelfRef':passNote that this is not a stupid conversion that is applied to all
annotations, it is only applied to annotations that are forward
references. Backward references are left as is.# Since 3.7classBackRef:defmethod(self)->ForwardRef:passclassForwardRef:defmethod(self)->BackRef:pass# From 3.0 to 3.6classBackRef:defmethod(self)->'ForwardRef':passclassForwardRef:defmethod(self)->BackRef:passIf you're supporting python 2.7, the annotation will of course be elided.PEP 498: formatted string literals.# Since 3.6who="World"print(f"Hello{who}!")# From 2.7 to 3.5who="World"print("Hello{0}!".format(who))The fixer also converts the newer{var=}syntax, even if you use
lib3to6 on a Python version older than 3.8.# Since 3.6who="World"print(f"Hello{who=}!")# From 2.7 to 3.5print("Hello who={0}!".format(who))Eliding of Annotations# Since 3.0deffoo(bar:int)->str:pass# In 2.7deffoo(bar):passPEP 515: underscores in numeric literals# Since 3.6num=1_234_567# From 2.7 to 3.5num=1234567Unpacking generalizationsFor literals...# Since 3.4x=[*[1,2],3]# From 2.7 to 3.3x=[1,2,3]For varargs...# Since 3.4foo(0,*a,*b)# From 2.7 to 3.3foo(*([0]+list(a)+list(b))For kwargs...# Since 3.4foo(**x,y=22,**z)# From 2.7 to 3.3importitertoolsfoo(**dict(itertools.chain(x.items(),{'y':22}.items(),z.items())))Note that the import will only be added to your module once.Keyword only arguments# Since 3.6defkwonly_func(*,kwonly_arg=1):...# From 2.7 to 3.5defkwonly_func(**kwargs):kwonly_arg=kwargs.get('kwonly_arg',1)...Convert class based typing.NamedTuple usage to assignmentsimporttyping# Since 3.5classBar(typing.NamedTuple):x:inty:str# From 2.7 to 3.4Bar=typing.NamedTuple('Bar',[('x',int),('y',str)])New Style Classes# Since 3.0classBar:pass# Before 3.0classBar(object):passFuture ImportsAll__future__imports applicable to your target version are
prepended to every file.# -*- coding: utf-8 -*-# This file is part of the <X> project# ..."""A docstring."""x=TrueWithtarget-version=27(the default).# -*- coding: utf-8 -*-# This file is part of the <X> project# ..."""A docstring."""from__future__importabsolute_importfrom__future__importdivisionfrom__future__importprint_functionfrom__future__importunicode_literalsx=TrueWithtarget-version=3.7# -*- coding: utf-8 -*-# This file is part of the <X> project# ..."""A docstring."""from__future__importannotationsx=TrueNote thatlib3to6works mostly at the ast level, but an exception is
made for any comments that appear at the top of the file. These are
preserved as is, so your shebang, file encoding and licensing headers
will be preserved.Not Supported FeaturesAn (obviously non-exhaustive) list of features which arenot
supported, either because they involve a semantic change, or
because there is no simple ast transformation to make them work
across different python versions:PEP 492 -async/awaitPEP 465 -@/__matmul__operatorPEP 380 -yield fromsyntaxPEP 584 - union operators fordictordered dictionary (since python 3.6)Modules with BackportsSome new modules have backports, which lib3to6 will point to:typingpathlib -> pathlib2secrets -> python2-secretsipaddress -> py2-ipaddresscsv -> backports.csvlzma -> backports.lzmaenum -> enum34For a full list of modules for which these warnings and errors apply,
please reviewMAYBE_UNUSABLE_MODULESin
src/lib3to6/checkers_backports.pyFor some modules, the backport uses the same module name as the original
module in the standard library. By default, lib3to6 will only warn about
usage of such modules, since it cannot detect if you're using the module
from the backported package (good) or from the standard library (bad if
not available in your target version). If you would like to opt-in to hard
error messages, you can whitelist modules for which you have the
backported package as a dependency.A good approach to adding such backports as dependencies is to
qualify the requirement with adependency
specification, so that
users with a newer interpreter use the builtin module and don't
install the backport package that they don't need.These work as arguments forinstall_requiresand also inrequirements.txtfiles.importsetuptoolssetuptools.setup(name="my-package",install_requires=['typing;python_version<"3.5"'],...)For testing, you can also pass these as a space separated parameter
to thelib3to6cli command:$lib3to6my_script.py>/dev/null
WARNING-my_script.py@1:Useofimport'enum'.ThismoduleisonlyavailablesincePython3.5,butyouconfiguredtarget_version=2.7.
WARNING-my_script.py@2:Useofimport'typing'.ThismoduleisonlyavailablesincePython3.5,butyouconfiguredtarget_version=2.7.

importenum
importtyping
...

$lib3to6`--install-requires='typing'`my_script.py>/dev/null
Traceback(mostrecentcalllast):...File"/home/user/.../lib3to6/src/lib3to6/checkers_backports.py",line134,in__call__raisecommon.CheckError(errmsg,node)lib3to6.common.CheckError:my_script.py@1-Prohibitedimport'enum'.ThismoduleisavailablesincePython3.4,butyouconfiguredtarget_version='2.7'.Use'https://pypi.org/project/enum34'instead.

$lib3to6`--install-requires='typing enum34'`my_script.py
importenum
importtyping
...MotivationThe main motivation for this project is to be able to usemypywithout sacrificing compatibility to older versions of python.# my_module/__init__.pydefhello(who:str)->None:importsysprint(f"Hello{who}from{sys.version.split()[0]}!")print(__file__)hello("世界")$pipinstalllib3to6
$python-mlib3to6my_module/__init__.py# -*- coding: utf-8 -*-from__future__importabsolute_importfrom__future__importdivisionfrom__future__importprint_functionfrom__future__importunicode_literalsdefhello(who):importsysprint('Hello{0}from{1}!'.format(who,sys.version.split()[0]))print(__file__)hello('世界')Fixes are applied to match the semantics of python3 code as
close as possible, even when running on a python2.7 interpreter.Some fixes that have been applied in the above:- PEP263 magic comment to declare the coding of the python
  source file. This allows the string literal `"世界"` to
  be decoded correctly.
- `__future__` imports have been added. This includes the well
  known print statement -> function change. The unicode_literals
- Type annotations have been removed
- `f""` string -> `"".format()` conversionCompatibility MattersI've seen a common gut reaction to lib3to6, which is that we shouldn't care about older versions of Python, Python 2.7 in particular. I would humbly suggest you consider the position of people other than developers who have full control over their development environment and only use CPython. As of this writing (August 2020), the most recent language version supported by alternative interpreters is the following:InterpreterVersionStackless3.7PyPy3.6MicroPython3.4IronPython2.7Jython2.7Notice that even Stackless Python, which has the least effort to keep up with new language features, is nonetheless lagging behind CPython. And even if all you care about is CPython, be aware that the most recent interpreter may not be available on platforms that users care about. For example on PythonAnywhere.com, the most recent version of CPython is 3.5 and of PyPy is 2.7.If you are writing a library and it doesn'tneedany of the newerruntimefeatures, such as async/await or ordered dictionaries, then I would humbly suggest you do not unnecessarily prevent users of such platforms from using your library.From a users perspective, only supporting the newest versions of Python might be interpreted as arrogance, but your time as a maintainer isn't free, and you don't owe users of your library anything. Lib3to6 exists to minimize your effort to maintain backward compatibility. If you have difficulties integrating lib3to6 into your packaging process, please report an issue:gitlab.com/mbarkhau/lib3to6/-/issues.On Testing your ProjectProjects that use lib3to6 should have a test-suite that is executed
with the oldest python version that you want to support, using the
converted output generated by lib3to6. While you can develop using a
newer version of python, you should not blindly trust lib3to6 as it is
very easy to introduce backward incompatible changes if you only test
on the most recent interpreter. The most obvious example is that
lib3to6 cannot do much to help you if a library producesbyteson
Python 2 butstron Python 3.The easiest way I have found to test a project, is to create a
distribution usingpython setup.py bdist_wheelwith the above
modifications to thesetup.py, install it and run the test-suite
against the installed modules.How it worksThis project works at the level of the python abstract syntax
tree (AST). The AST is transformed so that is only uses
constructs that are also valid in older versions of python. For
example it will translate f-strings to normal strings using thestr.formatmethod.>>>importsys>>>sys.version_info'3.6.5'>>>importlib3to6>>>py3_source='f"Hello {1 + 1}!"'>>>cfg={"fixers":["f_string_to_str_format"]}>>>py2_source=lib3to6.transpile_module(cfg,py3_source)>>>print(py3_source)f"Hello{1+1}!">>>print(py2_source)# -*- coding: utf-8 -*-"Hello{0}!".format(1+1)At a lower level, this translation is based on detection of theast.JoinedStrnode, which is translated into and AST that can be
serialized back into python syntax that will also work on older
versions.>>>print(lib3to6.parsedump_ast(py3_source))Module(body=[Expr(value=JoinedStr(values=[Str(s='Hello '),FormattedValue(value=BinOp(left=Num(n=1),op=Add(),right=Num(n=1)),conversion=-1,format_spec=None,),Str(s='!'),]))])>>>print(lib3to6.parsedump_ast(py2_source))Module(body=[Expr(value=Call(func=Attribute(value=Str(s='Hello{0}!'),attr='format',ctx=Load(),),args=[BinOp(left=Num(n=1),op=Add(),right=Num(n=1))],keywords=[]))])Checker ErrorsOf course this does not cover every aspect of compatibility.
Changes in APIs cannot be translated automatically in this way.An obvious example, is that there is no way to transpile code
which usesasyncandawait. In this case,lib3to6will simply raise a CheckError. This applies only to your source
code though, so if import use a library which usesasyncandawait, everything may look fine until you run your tests
on python 2.7.A more subtle example is the change in semantics of the builtinopenfunction.$catopen_example.py
withopen("myfile.txt",mode="w",encoding="utf-8")asfobj:fobj.write("Hello Wörld!")$python2open_example.py
Traceback(mostrecentcalllast):File"<string>",line1,in<module>
TypeError:'encoding'isaninvalidkeywordargumentforthisfunctionUsually there are alternative ways to write equivalent code that
works on all versions of python. For these common
incompatibilities lib3to6 will raise an error and suggest an
alternative, such as in this case usingio.openinstead.$lib3to6open_example.py
Traceback(Mostrecentcalllast):11lib3to6<module>-->sys.exit(main())764core.py__call__-->returnself.main(*args,**kwargs)717core.pymain-->rv=self.invoke(ctx)956core.pyinvoke-->returnctx.invoke(self.callback,**ctx.params)555core.pyinvoke-->returncallback(*args,**kwargs)55__main__.pymain-->fixed_source_text=transpile.transpile_module(cfg,source_text)260transpile.pytranspile_module-->checker(cfg,module_tree)158checkers.py__call__-->raisecommon.CheckError(msg,node)CheckError:Prohibitedkeywordargument'encoding'tobuiltin.open.online1ofopen_example.pyHerelib3to6you will give you aCheckError, however it
remains your responsibility to write your code so that this
syntactic translation is semantically equivalent in both python3
and python2.lib3to6uses the pythonastmodule to parse your code. This
means that you need a modern python interpreter to transpile from
modern python to legacy python interpreter. You cannot transpile
features which your interpreter cannot parse. The intended use is
for developers of libraries who use the most modern python
version, but want their libraries to work on older versions.ContributingThe most basic contribution you can make is to provide minimal,
reproducible examples of code that should either be converted or
which should raise an error.The project is hosted atgitlab.com/mbarkhau/lib3to6,
mainly because that's where the CI/CD is configured. GitHub is only
used as a copy/backup (and because that seems to be where many people
look for things).You can get started with local development in just a few commands.user@host:~/$gitclonehttps://gitlab.com/mbarkhau/lib3to6.git
user@host:~/$cdlib3to6/
user@host:~/lib3to6/⎇master$makehelpuser@host:~/lib3to6/⎇master$makeconda# creates conda environments...
user@host:~/lib3to6/⎇master$ls~/miniconda3/envs/
user@host:~/lib3to6_pypy35lib3to6_py27lib3to6_py36lib3to6_py37lib3to6_py38The targets in the makefile are set up to use the virtual environments.user@host:~/lib3to6/⎇master$makefmt
Alldone!✨🍰✨21filesleftunchanged.

user@host:~/lib3to6/⎇master$makelintmypydevtest
isort...ok
sjfmt...ok
flake8..ok
mypy....ok
...For debugging you may wish to activate a virtual environment anyway.user@host:~/lib3to6/⎇master$sourceactivate
user@host:~/lib3to6/⎇master(lib3to6_py38)$ipython
Python3.8.2|packagedbyconda-forge|(default,Apr242020,08:20:52)Type'copyright','credits'or'license'formoreinformation
IPython7.14.0--AnenhancedInteractivePython.Type'?'forhelp.

In[1]:importlib3to6

In[2]:lib3to6.__file__
Out[2]:'/home/user/lib3to6/src/lib3to6/__init__.py'Project Status (as of 2021-01-01): BetaI have tested with Python 3.9 with only minor updates required since Python 3.8. I've been using this library for over two years on a few projects
without much incident. Examples of such projects are:BumpVer.markdown-katexmarkdown-svgbobmarkdown-aafigurepylint-ignorepretty-tracebackbackports.pampyFuture WorkIn an ideal world, the project would cover all cases documented onhttp://python-future.organd either:Transpile to code that will work on any versionRaise an error, ideally pointing to a page and section on
python-future.org or other documentation describing
alternative methods of writing backwards compatible code.https://docs.python.org/3.X/whatsnew/also contains much info on
API changes that might be checked for, but checks and fixers for
these will only be written if they are common enough, otherwise
it's just too much work (patches are welcome though).AlternativesSince starting this project, I've learned of thepy-backwardsproject, which is
very, very similar in its approach. I have not evaluated it yet, to
determine for what projects it might be a better choice.Some features that might be implementedPEP 380 -yield from gensyntax might be supported in a basic form
by expanding to afor x in gen: yield x. That is not semantically
equivalent though and I don't know if it's worthimplementing it properlyPEP 465 -@operator could be done by replacing all cases
where the operator is used with a__matmul__method call.FAQQ: Isn't the tagline "Compatibility Matters" ironic,
considering that python 3.6+ is required to build a wheel?A: The irony is not lost. The issue is, how to parse source
code from a newer version of python than the python
interpreter itself supports. You can install lib3to6 on
older versions of python, but you'll be limited to the
features supported by that version. For example, you won't be
able to use f"" strings on python 3.5, but most annotations
will work fine.Q: Why keep python2.7 alive? Just let it die already!A: Indeed, and lib3to6 can help with that. Put yourself in the
shoes of somebody who is working on an old codebase. It's not
realistic hold all other development efforts while the
codebase is migrated and tested, while everything else waits.Instead an incremental approach is usually the only option.
With lib3to6, individual modules of the codebase can be
migrated to python3, leaving the rest of the codebase
untouched. The project can still run in a python 2.7
environment, while developers increasingly move to using
python 3.Additionally, lib3to6 is not just for compatibility with
python 2.7, it also allows you to use new features like f""
strings and variable annotations, while still maintaining
compatibility with older versions of python 3.Q: Why notlib3to2?A: I can't honestly say much aboutlib3to2. It seems to not
be maintained and looking at the source I thought it would be
easier to just write something new that worked on the AST level.
The scope oflib3to6is more general than 3to2, as you can
use it even if all you care about is converting from python 3.6
to 3.5.Changelog forhttps://github.com/mbarkhau/lib3to6v202107.1045Fix type annotations with list argumentsv202107.1044Fix type annotations with attribute accessv202101.1043Fixes for Python 3.9 and mypy 0.800v202009.1042New #6: per-file opt-in/opt-out using# lib3to6: disabled/# lib3to6: enabledv202006.1041New: Lots more documentation.New #5: Add detection of invalid imports and point to available backports. Useinstall_requiresoption to whitelist installed backports.New: Checkers produce better error messages.New: Colouring of diffs when usinglib3to6cli command.New: Checker foryield fromsyntax on target version doesn't support itNew: Checker for@operator when target version doesn't support itFix #3:--target-versionargument is ignoredgitlab../issues/3Fix #4: Removefrom __future__ import Xwhen the target version doesn't support it.Fix #4: Convert Forward Reference Annotations to stringsgitlab../issues/4Thank youFaidon Liambotisfor your help with testing and helping to debug ❤️.Fix: Don't apply keyword only args fixer for--target-version=3.0or above.v202002.0031Compatibility fixes for Python 3.8Add support for f-string=specifierAdd support for:=walrus operator (except inside comprehensions)v201902.0030Fix python 2 builtins were not always overridden correctly.Fix pypy compatibility testingBetter mypy coveragev201812.0021-betaRecursively apply some fixers.v201812.0020-alphaMove to gitlab.comUse bootstrapitFix bugs based on use with pycalverv201809.0019-alphaCheckErrors include log line numbersTranspile errors now include filenamesAdded fixers for renamed modules, e.g.
.. code-block:: diff- import queue
 + try:
 +     import queue
 + except ImportError:
 +     import Queue as queuev201808.0014-alphaBetter handling of package_dirChange toCalVer Versioning <https://calver.org/>_Remove console script in favour of simplepython -m lib3to6Rename fromthree2six->lib3to6