cloudmonkey CLI をpython3化してみる　−１

AWSやCloudnをAPIで利用できるようになるCLIツール[CloudStack cloudmonkey CLI](https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI)でも残念ながらPython2。そこで、勉強のためにPython3で動くようにしてみる。　

**まだ成功していません、メモとして利用中。**

# Python3化
ちょっと調べたところ一般的には、２パターンあり、引き続きpython2でも動くようにする、「Polyglot」と、それ以降はPython2をサポートしない「Port」があるらしい。

方向性
　　python2は使っていないので**Port**することにする。

# Python3化に関するドキュメント
[diveintopython](http://diveintopython3-ja.rdy.jp/porting-code-to-python-3-with-2to3.html)
# Python3化ツール
2から3に移行するのはよくあるみたいで、すぐに幾つかツールが見つかった。
[2to3](https://docs.python.org/3.5/library/2to3.html)
[Futurize](http://python-future.org/)
[Six](https://pythonhosted.org/six/)

# 作業履歴

## 作業用のディレクトリを作成してそこに移動

```sh
$ mkdir porting
$ cd porting
```
## 作業用の環境を準備して

```sh
$ pyvenv ven
$ source ven/bin/activate
```

ドキュメントには次のパッケージが必要だったのでインストール

```
readline
requests
Pygments
prettytable
 
argcomplete
```

```sh
$ pip install --upgrade readline requests Pygments prettytable argcomplete

$ pip freeze

```
argcomplete==1.4.1
prettytable==0.7.2
Pygments==2.1.3
readline==6.2.4.1
requests==2.11.1
```

それらに追加してポートするのにfutureをインストール
$ pip install future

## コードをクローンしてそこに移動
$ git clone https://github.com/apache/cloudstack-cloudmonkey.git
cd cloudstack-cloudmonkey

中身はこんな感じ

```sh
$ ls
```

```shell-session

CHANGES.md        LICENSE           NOTICE            cloudmonkey       docs              setup.cfg
Dockerfile        Makefile          README.md         config.docker     performrelease.sh setup.py
```

## 早速インストールしてみる
```sh
$ python setup.py develop
```


```
  File "setup.py", line 50
    print "If you're upgrading, run the following to enable parameter completion:"
                                                                                 ^
SyntaxError: Missing parentheses in call to 'print'
```

まあ、失敗しますね。
ドキュメンをは後で読み直すとして　、futurize --stage1　futurize --stage2って実行するとpython3 のコードに書き換えてくれるみたいなので。まずインストールできるようにsetup.pyをpytho3のコードに変換

$ futurize --stage1 -w setup.py

```
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored setup.py
--- setup.py   	(original)
+++ setup.py   	(refactored)
@@ -1,3 +1,4 @@
+from __future__ import print_function
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -47,10 +48,10 @@
         requires.append('readline')

 # Upgrade notes for 5.3.0
-print "If you're upgrading, run the following to enable parameter completion:"
-print "  cloudmonkey sync"
-print "  cloudmonkey set paramcompletion true"
-print "Parameter completion may fail, if the above is not run!"
+print("If you're upgrading, run the following to enable parameter completion:")
+print("  cloudmonkey sync")
+print("  cloudmonkey set paramcompletion true")
+print("Parameter completion may fail, if the above is not run!")

 setup(
     name = 'cloudmonkey',
RefactoringTool: Files that were modified:
RefactoringTool: setup.py
```

$ futurize --stage2 -w setup.py

```
RefactoringTool: Refactored setup.py
--- setup.py   	(original)
+++ setup.py   	(refactored)
@@ -1,4 +1,5 @@
 from __future__ import print_function
+from builtins import str
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
RefactoringTool: Files that were modified:
RefactoringTool: setup.py
```

作業が終わったので、インスールを実行したが
$ python setup.py develop

```
Traceback (most recent call last):
  File "setup.py", line 29, in <module>
    from cloudmonkey import __version__, __description__
  File "/Users/kentaro/porting/cloudstack-cloudmonkey/cloudmonkey/__init__.py", line 22
    except ImportError, e:
                      ^
SyntaxError: invalid syntax
```
__init__.pyがエラーになり、終了、今度は__init__.pyを変換してみる。
$ futurize --stage1 -w cloudmonkey/__init__.py

```
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored cloudmonkey/__init__.py
--- cloudmonkey/__init__.py    	(original)
+++ cloudmonkey/__init__.py    	(refactored)
@@ -1,3 +1,5 @@
+from __future__ import print_function
+from __future__ import absolute_import
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -16,8 +18,8 @@
 # under the License.

 try:
-    from config import __version__, __description__
-    from config import __maintainer__, __maintaineremail__
-    from config import __project__, __projecturl__, __projectemail__
-except ImportError, e:
-    print e
+    from .config import __version__, __description__
+    from .config import __maintainer__, __maintaineremail__
+    from .config import __project__, __projecturl__, __projectemail__
+except ImportError as e:
+    print(e)
RefactoringTool: Files that were modified:
RefactoringTool: cloudmonkey/__init__.py
```

$ futurize --stage2 -w cloudmonkey/__init__.py

```
RefactoringTool: No files need to be modified.
```
再度インストールしてみると。
$ python setup.py develop

```
Traceback (most recent call last):
  File "setup.py", line 29, in <module>
    from cloudmonkey import __version__, __description__
  File "/Users/kentaro/porting/cloudstack-cloudmonkey/cloudmonkey/__init__.py", line 21, in <module>
    from .config import __version__, __description__
  File "/Users/kentaro/porting/cloudstack-cloudmonkey/cloudmonkey/config.py", line 33
    except ImportError, e:
                      ^
SyntaxError: invalid syntax
```

今度はconfig.pyがエラーになり、終了、config.pyを変換してみる。
$ futurize --stage1 -w cloudmonkey/config.py

```
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored cloudmonkey/config.py
--- cloudmonkey/config.py      	(original)
+++ cloudmonkey/config.py      	(refactored)
@@ -16,6 +16,8 @@
 # specific language governing permissions and limitations
 # under the License.

+from __future__ import print_function
+from functools import reduce
 __version__ = "5.3.3"
 __description__ = "Command Line Interface for Apache CloudStack"
 __maintainer__ = "The Apache CloudStack Team"
@@ -30,8 +32,8 @@

     from ConfigParser import ConfigParser
     from os.path import expanduser
-except ImportError, e:
-    print "ImportError", e
+except ImportError as e:
+    print("ImportError", e)

 param_type = ['boolean', 'date', 'float', 'integer', 'short', 'list',
               'long', 'object', 'map', 'string', 'tzdate', 'uuid']
@@ -84,18 +86,18 @@
         try:
             with open(config_file, 'r') as cfg:
                 config.readfp(cfg)
-        except IOError, e:
-            print "Error: config_file not found", e
+        except IOError as e:
+            print("Error: config_file not found", e)

     profile = None
     try:
         profile = get_attr('profile')
-    except AttributeError, e:
+    except AttributeError as e:
         pass
     if profile is None or profile == '':
         profile = default_profile_name
     if profile in mandatory_sections:
-        print "Server profile name cannot be '%s'" % profile
+        print("Server profile name cannot be '%s'" % profile)
         sys.exit(1)

     has_profile_changed = False
@@ -117,8 +119,8 @@
                 else:
                     for key in config_fields[section].keys():
                         config.set(section, key, config_fields[section][key])
-            except ValueError, e:
-                print "Server profile name cannot be", profile
+            except ValueError as e:
+                print("Server profile name cannot be", profile)
                 sys.exit(1)
         if section in mandatory_sections:
             section_keys = config_fields[section].keys()
@@ -150,13 +152,13 @@
         try:
             with open(config_file, 'r') as cfg:
                 config.readfp(cfg)
-        except IOError, e:
-            print "Error: config_file not found", e
+        except IOError as e:
+            print("Error: config_file not found", e)
     else:
         config = write_config(get_attr, config_file)
-        print "Welcome! Use the `set` command to configure options"
-        print "Config file:", config_file
-        print "After setting up, run the `sync` command to sync apis\n"
+        print("Welcome! Use the `set` command to configure options")
+        print("Config file:", config_file)
+        print("After setting up, run the `sync` command to sync apis\n")

     missing_keys = []
     if config.has_option('core', 'profile'):
@@ -166,19 +168,19 @@
         profile = default_profile_name

     if profile is None or profile == '' or profile in mandatory_sections:
-        print "Server profile cannot be", profile
+        print("Server profile cannot be", profile)
         sys.exit(1)

     set_attr("profile_names", filter(lambda x: x != "core" and x != "ui",
                                      config.sections()))

     if not config.has_section(profile):
-        print ("Selected profile (%s) does not exist," +
-               " using default values") % profile
+        print(("Selected profile (%s) does not exist," +
+               " using default values") % profile)
         try:
             config.add_section(profile)
-        except ValueError, e:
-            print "Server profile name cannot be", profile
+        except ValueError as e:
+            print("Server profile name cannot be", profile)
             sys.exit(1)
         for key in default_profile.keys():
             config.set(profile, key, default_profile[key])
@@ -191,7 +193,7 @@
         for key in section_keys:
             try:
                 set_attr(key, config.get(section, key))
-            except Exception, e:
+            except Exception as e:
                 if section in mandatory_sections:
                     set_attr(key, config_fields[section][key])
                 else:
@@ -202,8 +204,8 @@
                 set_attr(key, get_attr('prompt').strip() + " ")

     if len(missing_keys) > 0:
-        print "Missing configuration was set using default values for keys:"
-        print "`%s` in %s" % (', '.join(missing_keys), config_file)
+        print("Missing configuration was set using default values for keys:")
+        print("`%s` in %s" % (', '.join(missing_keys), config_file))
         write_config(get_attr, config_file)

     return config_options
RefactoringTool: Files that were modified:
RefactoringTool: cloudmonkey/config.py
```

$ futurize --stage2 -w cloudmonkey/config.py

```
RefactoringTool: Refactored cloudmonkey/config.py
--- cloudmonkey/config.py      	(original)
+++ cloudmonkey/config.py      	(refactored)
@@ -17,6 +17,8 @@
 # under the License.

 from __future__ import print_function
+from future import standard_library
+standard_library.install_aliases()
 from functools import reduce
 __version__ = "5.3.3"
 __description__ = "Command Line Interface for Apache CloudStack"
@@ -30,7 +32,7 @@
     import os
     import sys

-    from ConfigParser import ConfigParser
+    from configparser import ConfigParser
     from os.path import expanduser
 except ImportError as e:
     print("ImportError", e)
@@ -114,18 +116,18 @@
             try:
                 config.add_section(section)
                 if section not in mandatory_sections:
-                    for key in default_profile.keys():
+                    for key in list(default_profile.keys()):
                         config.set(section, key, default_profile[key])
                 else:
-                    for key in config_fields[section].keys():
+                    for key in list(config_fields[section].keys()):
                         config.set(section, key, config_fields[section][key])
             except ValueError as e:
                 print("Server profile name cannot be", profile)
                 sys.exit(1)
         if section in mandatory_sections:
-            section_keys = config_fields[section].keys()
+            section_keys = list(config_fields[section].keys())
         else:
-            section_keys = default_profile.keys()
+            section_keys = list(default_profile.keys())
         for key in section_keys:
             try:
                 if not (has_profile_changed and section == profile):
@@ -143,9 +145,8 @@
     if not os.path.exists(config_dir):
         os.makedirs(config_dir)

-    config_options = reduce(lambda x, y: x + y, map(lambda x:
-                            config_fields[x].keys(), config_fields.keys()))
-    config_options += default_profile.keys()
+    config_options = reduce(lambda x, y: x + y, [list(config_fields[x].keys()) for x in list(config_fields.keys())])
+    config_options += list(default_profile.keys())

     config = ConfigParser()
     if os.path.exists(config_file):
@@ -171,8 +172,7 @@
         print("Server profile cannot be", profile)
         sys.exit(1)

-    set_attr("profile_names", filter(lambda x: x != "core" and x != "ui",
-                                     config.sections()))
+    set_attr("profile_names", [x for x in config.sections() if x != "core" and x != "ui"])

     if not config.has_section(profile):
         print(("Selected profile (%s) does not exist," +
@@ -182,14 +182,14 @@
         except ValueError as e:
             print("Server profile name cannot be", profile)
             sys.exit(1)
-        for key in default_profile.keys():
+        for key in list(default_profile.keys()):
             config.set(profile, key, default_profile[key])

     for section in (mandatory_sections + [profile]):
         if section in mandatory_sections:
-            section_keys = config_fields[section].keys()
+            section_keys = list(config_fields[section].keys())
         else:
-            section_keys = default_profile.keys()
+            section_keys = list(default_profile.keys())
         for key in section_keys:
             try:
                 set_attr(key, config.get(section, key))
RefactoringTool: Files that were modified:
RefactoringTool: cloudmonkey/config.py
```
インストールに再度挑戦
$ python setup.py develop

```
If you're upgrading, run the following to enable parameter completion:
  cloudmonkey sync
  cloudmonkey set paramcompletion true
Parameter completion may fail, if the above is not run!
running develop
running egg_info
creating cloudmonkey.egg-info
writing dependency_links to cloudmonkey.egg-info/dependency_links.txt
writing cloudmonkey.egg-info/PKG-INFO
writing entry points to cloudmonkey.egg-info/entry_points.txt
writing top-level names to cloudmonkey.egg-info/top_level.txt
writing requirements to cloudmonkey.egg-info/requires.txt
writing manifest file 'cloudmonkey.egg-info/SOURCES.txt'
reading manifest file 'cloudmonkey.egg-info/SOURCES.txt'
writing manifest file 'cloudmonkey.egg-info/SOURCES.txt'
running build_ext
Creating /Users/kentaro/porting/ven/lib/python3.5/site-packages/cloudmonkey.egg-link (link to .)
Adding cloudmonkey 5.3.3 to easy-install.pth file
Installing cloudmonkey script to /Users/kentaro/porting/ven/bin

Installed /Users/kentaro/porting/cloudstack-cloudmonkey
Processing dependencies for cloudmonkey==5.3.3
Searching for requests-toolbelt
Reading https://pypi.python.org/simple/requests-toolbelt/
Best match: requests-toolbelt 0.7.0
Downloading https://pypi.python.org/packages/59/78/1d391d30ebf74079a8e4de6ab66fdca5362903ef2df64496f4697e9bb626/requests-toolbelt-0.7.0.tar.gz#md5=bfe2009905f460f4764c32cfbbf4205f
Processing requests-toolbelt-0.7.0.tar.gz
Writing /var/folders/vl/nj7qbspd7hlgll3chjfgvl400000gn/T/easy_install-l6qjdcgd/requests-toolbelt-0.7.0/setup.cfg
Running requests-toolbelt-0.7.0/setup.py -q bdist_egg --dist-dir /var/folders/vl/nj7qbspd7hlgll3chjfgvl400000gn/T/easy_install-l6qjdcgd/requests-toolbelt-0.7.0/egg-dist-tmp-xbz2yomj
no previously-included directories found matching 'docs/_build'
warning: no previously-included files matching '*.py[cdo]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
zip_safe flag not set; analyzing archive contents...
Copying requests_toolbelt-0.7.0-py3.5.egg to /Users/kentaro/porting/ven/lib/python3.5/site-packages
Adding requests-toolbelt 0.7.0 to easy-install.pth file

Installed /Users/kentaro/porting/ven/lib/python3.5/site-packages/requests_toolbelt-0.7.0-py3.5.egg
Searching for dicttoxml
Reading https://pypi.python.org/simple/dicttoxml/
Best match: dicttoxml 1.7.4
Downloading https://pypi.python.org/packages/74/36/534db111db9e7610a41641a1f6669a964aacaf51858f466de264cc8dcdd9/dicttoxml-1.7.4.tar.gz#md5=ec5643a048cf32dad3c28db236b923e4
Processing dicttoxml-1.7.4.tar.gz
Writing /var/folders/vl/nj7qbspd7hlgll3chjfgvl400000gn/T/easy_install-3yyvt_ku/dicttoxml-1.7.4/setup.cfg
Running dicttoxml-1.7.4/setup.py -q bdist_egg --dist-dir /var/folders/vl/nj7qbspd7hlgll3chjfgvl400000gn/T/easy_install-3yyvt_ku/dicttoxml-1.7.4/egg-dist-tmp-1oj02owk
zip_safe flag not set; analyzing archive contents...
Copying dicttoxml-1.7.4-py3.5.egg to /Users/kentaro/porting/ven/lib/python3.5/site-packages
Adding dicttoxml 1.7.4 to easy-install.pth file

Installed /Users/kentaro/porting/ven/lib/python3.5/site-packages/dicttoxml-1.7.4-py3.5.egg
Searching for requests==2.11.1
Best match: requests 2.11.1
Adding requests 2.11.1 to easy-install.pth file

Using /Users/kentaro/porting/ven/lib/python3.5/site-packages
Searching for prettytable==0.7.2
Best match: prettytable 0.7.2
Adding prettytable 0.7.2 to easy-install.pth file

Using /Users/kentaro/porting/ven/lib/python3.5/site-packages
Searching for argcomplete==1.4.1
Best match: argcomplete 1.4.1
Adding argcomplete 1.4.1 to easy-install.pth file

Using /Users/kentaro/porting/ven/lib/python3.5/site-packages
Searching for Pygments==2.1.3
Best match: Pygments 2.1.3
Adding Pygments 2.1.3 to easy-install.pth file
Installing pygmentize script to /Users/kentaro/porting/ven/bin

Using /Users/kentaro/porting/ven/lib/python3.5/site-packages
Finished processing dependencies for cloudmonkey==5.3.3
```

インストールに成功した。

$ cloudmonkey

```
Traceback (most recent call last):
  File "/Users/kentaro/porting/ven/bin/cloudmonkey", line 9, in <module>
    load_entry_point('cloudmonkey', 'console_scripts', 'cloudmonkey')()
  File "/Users/kentaro/porting/ven/lib/python3.5/site-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/kentaro/porting/ven/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/Users/kentaro/porting/ven/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/Users/kentaro/porting/ven/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/kentaro/porting/cloudstack-cloudmonkey/cloudmonkey/cloudmonkey.py", line 48
    except ImportError, e:
                      ^
SyntaxError: invalid syntax
```

でもエラーになって終了しました。

今日はここまで。