Repository: networkx/networkx

Issue:
xrange vs python 3
Searching the networkx code for `xrange` I see it's used in some "Shapefile" related code and tests.  Should this be updated for python 3 compatibility, and is it not tested in the TravisCI testing?



Discussion:
There is also an 'iteritems" in there and maybe some other Python2isms. I suggest we write some proper tests for that module or else consider removing it.

Here is an nice example that uses read_shp(), http://ipython-books.github.io/featured-03.html

From that link: "At the time of this writing, NetworkX's support of Shapefile doesn't seem to be compatible with Python 3.x."

Yes, but it is an indication that there are users and making it Python3
compatible would be good.  I wonder if the osgeo library this uses is
Python3 ready?

On Tue, Jul 29, 2014 at 8:32 PM, argriffing notifications@github.com
wrote:

> From that link: "At the time of this writing, NetworkX's support of
> Shapefile doesn't seem to be compatible with Python 3.x."
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/networkx/networkx/issues/1227#issuecomment-50566825.

>  I wonder if the osgeo library this uses is Python3 ready?

Yes according to http://trac.osgeo.org/gdal/wiki/GdalOgrInPython it's been ready since version https://pypi.python.org/pypi/GDAL/1.7.0 which was uploaded in early 2010.  The .shp files are binary (http://en.wikipedia.org/wiki/Shapefile#Shapefile_shape_format_.28.shp.29), so not easy to paste directly into python unit testing code.

Maybe @bwreilly (original author) can help us update this and figure out how to test it? 

I think testing it should be about the [same as it is now](https://github.com/networkx/networkx/blob/master/networkx/readwrite/tests/test_shp.py), which is to say a pair of shapefiles are created temporarily as part of `setUp`, used for testing, and then eliminated in the `tearDown` method.

As for updating to Python 3, I could take a look at it over the weekend. But I'll admit I haven't done much Python 3 in general.

Ah yes, there are tests.  I completely missed that.  Updating to Python3 should be pretty simple (xrange->range, iteritems()->items() etc).  I'll take a look and see if there are other issues.

I tried fixing those two issues (xrange,iteritems) and the tests fail - maybe the file isn't getting loaded correctly?
So any help you can give would be great.

The tests may not need to change much, but unless the environment has `osgeo` they will be skipped.  For Ubuntu it is packaged under `python3-gdal` if that would help a Python3 TravisCI configuration find it.

Yes, that is the correct package (python-gdal, python3-gdal).  

Added Travis testing under Python 2.7 in dda332e. Had to work around travis-ci/travis-ci#2683 and force `pip` to install an older version of GDAL because the latest is too new for the `libgdal-dev` from `apt-get` on Travis. For Python 3.x, the necessary packages do install, but `f.geometry()` in the code returns `None` instead of something meaningful and breaks the tests as a result.

This needs someone familar with the GDAL bindings to take a look (at minimum the `f.geometry()` issue).


Resolving patch:
diff --git a/networkx/readwrite/nx_shp.py b/networkx/readwrite/nx_shp.py
--- a/networkx/readwrite/nx_shp.py
+++ b/networkx/readwrite/nx_shp.py
@@ -55,16 +55,15 @@ def read_shp(path):
     except ImportError:
         raise ImportError("read_shp requires OGR: http://www.gdal.org/")
 
-    net = nx.DiGraph()
-
-    def getfieldinfo(lyr, feature, flds):
-            f = feature
-            return [f.GetField(f.GetFieldIndex(x)) for x in flds]
+    if not isinstance(path, str):
+        return
 
-    def addlyr(lyr, fields):
-        for findex in range(lyr.GetFeatureCount()):
-            f = lyr.GetFeature(findex)
-            flddata = getfieldinfo(lyr, f, fields)
+    net = nx.DiGraph()
+    shp = ogr.Open(path)
+    for lyr in shp:
+        fields = [x.GetName() for x in lyr.schema]
+        for f in lyr:
+            flddata = [f.GetField(f.GetFieldIndex(x)) for x in fields]
             g = f.geometry()
             attributes = dict(zip(fields, flddata))
             attributes["ShpName"] = lyr.GetName()
@@ -76,14 +75,6 @@ def addlyr(lyr, fields):
                 attributes["Json"] = g.ExportToJson()
                 last = g.GetPointCount() - 1
                 net.add_edge(g.GetPoint_2D(0), g.GetPoint_2D(last), attributes)
-
-    if isinstance(path, str):
-        shp = ogr.Open(path)
-        lyrcount = shp.GetLayerCount()  # multiple layers indicate a directory
-        for lyrindex in range(lyrcount):
-            lyr = shp.GetLayerByIndex(lyrindex)
-            flds = [x.GetName() for x in lyr.schema]
-            addlyr(lyr, flds)
     return net
 
 
@@ -168,7 +159,7 @@ def create_feature(geometry, lyr, attributes=None):
         pass
     nodes = shpdir.CreateLayer("nodes", None, ogr.wkbPoint)
     for n in G:
-        data = G.node[n] or {}
+        data = G.node[n]
         g = netgeometry(n, data)
         create_feature(g, nodes)
     try:


Test patch:
diff --git a/networkx/readwrite/tests/test_shp.py b/networkx/readwrite/tests/test_shp.py
--- a/networkx/readwrite/tests/test_shp.py
+++ b/networkx/readwrite/tests/test_shp.py
@@ -50,7 +50,8 @@ def createlayer(driver):
         for path, name in zip(self.paths, self.names):
             feat = ogr.Feature(lyr.GetLayerDefn())
             g = ogr.Geometry(ogr.wkbLineString)
-            map(lambda xy: g.AddPoint_2D(*xy), path)
+            for p in path:
+                g.AddPoint_2D(*p)
             feat.SetGeometry(g)
             feat.SetField("Name", name)
             lyr.CreateFeature(feat)
@@ -60,7 +61,8 @@ def createlayer(driver):
 
     def testload(self):
         expected = nx.DiGraph()
-        map(expected.add_path, self.paths)
+        for p in self.paths:
+            expected.add_path(p)
         G = nx.read_shp(self.shppath)
         assert_equal(sorted(expected.node), sorted(G.node))
         assert_equal(sorted(expected.edges()), sorted(G.edges()))
@@ -101,7 +103,7 @@ def testattributes(lyr, graph):
             while feature:
                 coords = []
                 ref = feature.GetGeometryRef()
-                for i in xrange(ref.GetPointCount()):
+                for i in range(ref.GetPointCount()):
                     coords.append(ref.GetPoint_2D(i))
                 name = feature.GetFieldAsString('Name')
                 assert_equal(graph.get_edge_data(*coords)['Name'], name)