Repository: networkx/networkx

Issue:
nx.from_pandas_edgelist() iterates over DataFrame.values which changes dtypes
I arrived here through:

```
>>> df = pd.DataFrame([[1, 2, '0.1']], columns=['source', 'target', 'other'])
>>> nx.from_pandas_edgelist(df).nodes()
NodeView((1, 2))

>>> df = pd.DataFrame([[1, 2, 0.1]], columns=['source', 'target', 'other'])
>>> nx.from_pandas_edgelist(df).nodes()
NodeView((1.0, 2.0))
```

I initially thought this would be an issue with [`from_pandas_edgelist`](https://github.com/networkx/networkx/blob/9ac4941c1cb25bf47c85d2bb19f7557f5c807694/networkx/convert_matrix.py#L241) but that code just hands off directly to add_edge:

https://github.com/networkx/networkx/blob/9ac4941c1cb25bf47c85d2bb19f7557f5c807694/networkx/convert_matrix.py#L346-L350

which means that [Graph().add_edge](https://github.com/networkx/networkx/blob/e97abea688a471a8422a676ea7c93d079bfa0f1f/networkx/classes/graph.py#L812-L874) must be responsible.

This is in networkx version 2.1




Discussion:
Update: This is happening before `Graph.add_edge()`

I changed the lines in convert_matrix to:
```
    else:
        for row in df.values:
            print(row)
            print(row[src_i])
            print(row[tar_i])
            print(type(row[src_i]))
            print(type(row[tar_i]))
            g.add_edge(row[src_i], row[tar_i])
```

and I'm seeing the types change before `g.add_edge`. That means this is an issue with `from_pandas_edgelist` or maybe... (gasp) an issue with pandas? 
This "feature" is documented in both networkx and pandas. It involves ```DataFrame.values``` using a flexible type for numeric values.   From [the docstring for](https://networkx.github.io/documentation/stable/reference/generated/networkx.convert_matrix.from_pandas_edgelist.html) ```nx.from_pandas_edgelist```:

    Note: This function iterates over DataFrame.values, which is not
    guaranteed to retain the data type across columns in the row. This is only
    a problem if your row is entirely numeric and a mix of ints and floats. In
    that case, all values will be returned as floats. See the
    DataFrame.iterrows documentation for an example.
Thanks @dschult! I should have read the docs better. Could you point me towards the pandas documentation of this feature? 
And could you explain why networkx iterates over `.values`? the docstring only lists the disadvantage. It seems like my issue could be fixed by iterating over the dataframe another way than `.values`
Ahh, I see the pandas docs: It's documented right with [df.values()](http://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.DataFrame.values.html)

I'd still love to hear what motivated this choice for the purposes of `from_pandas_edgelist`
@dschult why does networkx iterate over `.values`? 
That's the way to iterate over the matrix of values in the dataframe. Other suggestions? 
@dschult how about [`itertuples`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.itertuples.html)? 

```
n = 100000  # 100k edges
df = pd.concat([pd.DataFrame(np.random.randint(0,1000,(n,2)), columns=['source', 'target']), pd.DataFrame(np.random.random(n))], axis=1)
```

```
%%time
nx.from_pandas_edgelist(df)

CPU times: user 243 ms, sys: 15.2 ms, total: 258 ms
Wall time: 256 ms
```

```
%%time
g = nx.Graph()
for row in df.itertuples():
    g.add_edge(row[0], row[1])

CPU times: user 696 ms, sys: 49.2 ms, total: 746 ms
Wall time: 744 ms
```

Slower, but solves this dtype issue. 
1M edges: 
```
CPU times: user 2.76 s, sys: 83.7 ms, total: 2.85 s
Wall time: 2.85 s
```
vs
```
CPU times: user 6.9 s, sys: 330 ms, total: 7.23 s
Wall time: 7.23 s
```
And 10M edges:
```
CPU times: user 28.1 s, sys: 211 ms, total: 28.3 s
Wall time: 28.3 s
```
vs
```
CPU times: user 1min 13s, sys: 7.86 s, total: 1min 21s
Wall time: 1min 22s
```
Looks like a consistent factor of 3. 
@dschult here's a slightly messy, but faster way to do it: 
```
n = 1000000  # 1M edges
df = pd.DataFrame(np.random.randint(0,1000,(n,2)),columns=['source','target'])
```
```
%%time
nx.from_pandas_edgelist(df)
```
```
CPU times: user 2.58 s, sys: 59.5 ms, total: 2.64 s
Wall time: 2.64 s
```
```
%%time
g = nx.Graph()
[g.add_edge(s,t) for s,t in zip(df['source'], df['target'])]
```
```
CPU times: user 2.03 s, sys: 82.8 ms, total: 2.12 s
Wall time: 2.12 s
```



You shifted from testing that it worked to testing its speed.  But assuming all the above works, could you put together a PR with the improved version?  Thanks!


Resolving patch:
diff --git a/networkx/convert_matrix.py b/networkx/convert_matrix.py
--- a/networkx/convert_matrix.py
+++ b/networkx/convert_matrix.py
@@ -182,8 +182,9 @@ def from_pandas_adjacency(df, create_using=None):
     try:
         df = df[df.index]
     except:
-        raise nx.NetworkXError("Columns must match Indices.", "%s not in columns" %
-                               list(set(df.index).difference(set(df.columns))))
+        msg = "%s not in columns"
+        missing = list(set(df.index).difference(set(df.columns)))
+        raise nx.NetworkXError("Columns must match Indices.", msg % missing)
 
     nx.relabel.relabel_nodes(G, dict(enumerate(df.columns)), copy=False)
     return G
@@ -233,7 +234,8 @@ def to_pandas_edgelist(G, source='source', target='target', nodelist=None,
     source_nodes = [s for s, t, d in edgelist]
     target_nodes = [t for s, t, d in edgelist]
     all_keys = set().union(*(d.keys() for s, t, d in edgelist))
-    edge_attr = {k: [d.get(k, float("nan")) for s, t, d in edgelist] for k in all_keys}
+    edge_attr = {k: [d.get(k, float("nan")) for s, t, d in edgelist]
+                 for k in all_keys}
     edgelistdict = {source: source_nodes, target: target_nodes}
     edgelistdict.update(edge_attr)
     return pd.DataFrame(edgelistdict)
@@ -268,8 +270,8 @@ def from_pandas_edgelist(df, source='source', target='target', edge_attr=None,
 
     edge_attr : str or int, iterable, True
         A valid column name (str or integer) or list of column names that will
-        be used to retrieve items from the row and add them to the graph as edge
-        attributes. If `True`, all of the remaining columns will be added.
+        be used to retrieve items from the row and add them to the graph as
+        edge attributes. If `True`, all of the remaining columns will be added.
 
     create_using : NetworkX graph constructor, optional (default=nx.Graph)
        Graph type to create. If graph instance, then cleared before populated.
@@ -311,43 +313,34 @@ def from_pandas_edgelist(df, source='source', target='target', edge_attr=None,
     'red'
 
     """
-
     g = nx.empty_graph(0, create_using)
 
-    # Index of source and target
-    src_i = df.columns.get_loc(source)
-    tar_i = df.columns.get_loc(target)
-    if edge_attr:
-        # If all additional columns requested, build up a list of tuples
-        # [(name, index),...]
-        if edge_attr is True:
-            # Create a list of all columns indices, ignore nodes
-            edge_i = []
-            for i, col in enumerate(df.columns):
-                if col is not source and col is not target:
-                    edge_i.append((col, i))
-        # If a list or tuple of name is requested
-        elif isinstance(edge_attr, (list, tuple)):
-            edge_i = [(i, df.columns.get_loc(i)) for i in edge_attr]
-        # If a string or int is passed
-        else:
-            edge_i = [(edge_attr, df.columns.get_loc(edge_attr)), ]
-
-        # Iteration on values returns the rows as Numpy arrays
-        for row in df.values:
-            s, t = row[src_i], row[tar_i]
-            if g.is_multigraph():
-                g.add_edge(s, t)
-                key = max(g[s][t])  # default keys just count, so max is most recent
-                g[s][t][key].update((i, row[j]) for i, j in edge_i)
-            else:
-                g.add_edge(s, t)
-                g[s][t].update((i, row[j]) for i, j in edge_i)
-
-    # If no column names are given, then just return the edges.
+    if edge_attr is None:
+        g.add_edges_from(zip(df[source], df[target]))
+        return g
+
+    # Additional columns requested
+    if edge_attr is True:
+        cols = [c for c in df.columns if c is not source and c is not target]
+    elif isinstance(edge_attr, (list, tuple)):
+        cols = edge_attr
     else:
-        for row in df.values:
-            g.add_edge(row[src_i], row[tar_i])
+        cols = [edge_attr]
+
+    try:
+        eattrs = zip(*[df[col] for col in cols])
+    except (KeyError, TypeError) as e:
+        msg = "Invalid edge_attr argument: %s" % edge_attr
+        raise nx.NetworkXError(msg)
+    for s, t, attrs in zip(df[source], df[target], eattrs):
+
+        g.add_edge(s, t)
+
+        if g.is_multigraph():
+            key = max(g[s][t])  # default keys just count so max is most recent
+            g[s][t][key].update((attr, val) for attr, val in zip(cols, attrs))
+        else:
+            g[s][t].update((attr, val) for attr, val in zip(cols, attrs))
 
     return g
 
@@ -467,10 +460,10 @@ def from_numpy_matrix(A, parallel_edges=False, create_using=None):
         An adjacency matrix representation of a graph
 
     parallel_edges : Boolean
-        If this is True, `create_using` is a multigraph, and `A` is an
+        If True, `create_using` is a multigraph, and `A` is an
         integer matrix, then entry *(i, j)* in the matrix is interpreted as the
-        number of parallel edges joining vertices *i* and *j* in the graph. If it
-        is False, then the entries in the adjacency matrix are interpreted as
+        number of parallel edges joining vertices *i* and *j* in the graph.
+        If False, then the entries in the adjacency matrix are interpreted as
         the weight of a single edge joining the vertices.
 
     create_using : NetworkX graph constructor, optional (default=nx.Graph)


Test patch:
diff --git a/networkx/tests/test_convert_pandas.py b/networkx/tests/test_convert_pandas.py
--- a/networkx/tests/test_convert_pandas.py
+++ b/networkx/tests/test_convert_pandas.py
@@ -58,6 +58,13 @@ def test_from_edgelist_multi_attr(self):
         G = nx.from_pandas_edgelist(self.df, 0, 'b', ['weight', 'cost'])
         assert_graphs_equal(G, Gtrue)
 
+    def test_from_edgelist_multi_attr_incl_target(self):
+        Gtrue = nx.Graph([('E', 'C', {0: 'C', 'b': 'E', 'weight': 10}),
+                          ('B', 'A', {0: 'B', 'b': 'A', 'weight': 7}),
+                          ('A', 'D', {0: 'A', 'b': 'D', 'weight': 4})])
+        G = nx.from_pandas_edgelist(self.df, 0, 'b', [0, 'b', 'weight'])
+        assert_graphs_equal(G, Gtrue)
+
     def test_from_edgelist_multidigraph_and_edge_attr(self):
         # example from issue #2374
         Gtrue = nx.MultiDiGraph([('X1', 'X4', {'Co': 'zA', 'Mi': 0, 'St': 'X1'}),
@@ -94,6 +101,23 @@ def test_from_edgelist_one_attr(self):
         G = nx.from_pandas_edgelist(self.df, 0, 'b', 'weight')
         assert_graphs_equal(G, Gtrue)
 
+    def test_from_edgelist_int_attr_name(self):
+        # note: this also tests that edge_attr can be `source`
+        Gtrue = nx.Graph([('E', 'C', {0: 'C'}),
+                          ('B', 'A', {0: 'B'}),
+                          ('A', 'D', {0: 'A'})])
+        G = nx.from_pandas_edgelist(self.df, 0, 'b', 0)
+        assert_graphs_equal(G, Gtrue)
+
+    def test_from_edgelist_invalid_attr(self):
+        assert_raises(nx.NetworkXError, nx.from_pandas_edgelist,
+                      self.df, 0, 'b', 'misspell')
+        assert_raises(nx.NetworkXError, nx.from_pandas_edgelist,
+                      self.df, 0, 'b', 1)
+        # unhashable attribute name
+        assert_raises(nx.NetworkXError, nx.from_pandas_edgelist,
+                      self.df, 0, 'b', {})
+
     def test_from_edgelist_no_attr(self):
         Gtrue = nx.Graph([('E', 'C', {}),
                           ('B', 'A', {}),