Repository: pandas-dev/pandas

Issue:
concat produces incorrect output
Under certain circumstances, concat seems to produce erroneous results. I haven't worked out what causes the problems to arise, but here's an example:

df1 = DataFrame({'firmNo' : [0,0,0,0], 'stringvar' : ['rrr', 'rrr', 'rrr', 'rrr'], 'prc' : [6,6,6,6] })
df2 = DataFrame({'misc' : [1,2,3,4], 'prc' : [6,6,6,6], 'C' : [9,10,11,12]})
concat([df1,df2],axis=1)

produces as output:

  firmNo  prc  stringvar   C  misc  prc
0    rrr    0          6   9     1    6
1    rrr    0          6  10     2    6
2    rrr    0          6  11     3    6
3    rrr    0          6  12     4    6



Discussion:
@rhstanton It's helpful if you can put your output in "``" so that it prints in a monospaced font. It's easier on the eyes. :) I can reproduce this on git master. What is the expected output?

Ah. looks like there's a sorting problem here...

I agree it looks terrible! Does the output go in quotes in my notebook or when I upload to github? If you could give me a quick example of how to do this, I'd be more than happy to help others' eyesight in future.

From: Phillip Cloud <notifications@github.com<mailto:notifications@github.com>>
Reply-To: pydata/pandas <reply@reply.github.com<mailto:reply@reply.github.com>>
Date: Wednesday, May 15, 2013 3:42 PM
To: pydata/pandas <pandas@noreply.github.com<mailto:pandas@noreply.github.com>>
Cc: Richard Stanton <stanton@haas.berkeley.edu<mailto:stanton@haas.berkeley.edu>>
Subject: Re: [pandas] concat produces incorrect output (#3602)

@rhstantonhttps://github.com/rhstanton It's helpful if you can put your output in "``" so that it prints in a monospaced fonts. It's easier on the eyes. :)

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3602#issuecomment-17970916.

Surround anything you want monospaced type with backquotes, i.e., the **`** character. For example `x = 1`. When you get a chance to get on GitHub (not re ing from ur email) you should click on the GitHub Flavored Markdown link. It's full of useful info.

@cpcloud any luck with this?

Nah not yet, but I haven't given it more than a cursory glance. I will look
into it this weekend.
On May 17, 2013 9:02 PM, "jreback" notifications@github.com wrote:

> @cpcloud https://github.com/cpcloud any luck with this?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3602#issuecomment-18092899
> .

@jreback @rhstanton What is expected output? 2 `prc` columns with the same values? one of the `merge` behaviors?

Is this the expected output? (I will assume it is since it is the least magical thing `concat` could do here: just basically "join" [not in the database sense] the two frames together along the requested `axis`.)

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>firmNo</th>
      <th>prc</th>
      <th>stringvar</th>
      <th>C</th>
      <th>misc</th>
      <th>prc</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td> 0</td>
      <td> 6</td>
      <td> rrr</td>
      <td>  9</td>
      <td> 1</td>
      <td> 6</td>
    </tr>
    <tr>
      <th>1</th>
      <td> 0</td>
      <td> 6</td>
      <td> rrr</td>
      <td> 10</td>
      <td> 2</td>
      <td> 6</td>
    </tr>
    <tr>
      <th>2</th>
      <td> 0</td>
      <td> 6</td>
      <td> rrr</td>
      <td> 11</td>
      <td> 3</td>
      <td> 6</td>
    </tr>
    <tr>
      <th>3</th>
      <td> 0</td>
      <td> 6</td>
      <td> rrr</td>
      <td> 12</td>
      <td> 4</td>
      <td> 6</td>
    </tr>
  </tbody>
</table>

(heh i will try to parse this with `read_html` later...)

Yes, I’d expect the output you show below, just with the right column headings (2 prc columns with the same values, but only because they were passed in with the same values. If they’d had different values in df1 and df2, I’d expect two prc columns with different contents).

Best,

Richard

From: Phillip Cloud [mailto:notifications@github.com]
Sent: Saturday, May 18, 2013 6:42 PM
To: pydata/pandas
Cc: Richard Stanton
Subject: Re: [pandas] concat produces incorrect output (#3602)

@jrebackhttps://github.com/jreback @rhstantonhttps://github.com/rhstanton What is expected output? 2 prc columns with the same values? one of the merge behaviors? if u do

# use dfs from above

df = concat([df1, df2], axis, ignore_index=True)

print df

0

1

2

3

4

5

0

0

6

Rrr

9

1

6

1

0

6

Rrr

10

2

6

2

0

6

Rrr

11

3

6

3

0

6

Rrr

12

4

6

Is this what u want except with the original column indices?
(heh i will try to parse this with read_html later...)

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3602#issuecomment-18110879.

@jreback This is a strange beast i went all the way into ndframe.init and back up to concat. values attrs test the same, prolly a repr bug now

nvm something else...

@jreback AH HA! the bug is that the `_ref_locs` attribute of `BlockManager` is not set when u `concat` the two dfs and keep track of the index, but when u ignore the index and then set the columns there is already an ordering (_ref_locs is already set) so u r good. the question remains tho how u want to deal with this...seems like might want to raise when trying to concat in this situation. right now the exception thrown by get_indexer is caught and the assumption is made that the ordering is 0..n - 1 where n is the number of blocks, but that doesn't seem totally consistent, not sure what the optimal approach here is.

828f9f99 fixed the series version of this by just assigning the columns after the concat. is that the correct fix here? don't think so, maybe can ignore index if ignore_index is false and there are dup cols and axis is 1

Looks like that would work given the results of your earlier concat without column names.

From: Phillip Cloud [mailto:notifications@github.com]
Sent: Saturday, May 18, 2013 11:01 PM
To: pydata/pandas
Cc: Richard Stanton
Subject: Re: [pandas] concat produces incorrect output (#3602)

828f9f9https://github.com/pydata/pandas/commit/828f9f99 fixed the series version of this by just assigning the columns after the concat. is that the correct fix here?

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3602#issuecomment-18112898.

u might be right, need to see how to do this in a sane way...

i wonder if an `__eq__` on block and blkmgr might help things like this in the future. could compare items, values and ref_locs

let me take a look


Resolving patch:
diff --git a/pandas/tools/merge.py b/pandas/tools/merge.py
--- a/pandas/tools/merge.py
+++ b/pandas/tools/merge.py
@@ -1043,6 +1043,7 @@ def _concat_blocks(self, blocks):
                                 'DataFrames')
             return make_block(concat_values, blocks[0].items, self.new_axes[0])
         else:
+
             offsets = np.r_[0, np.cumsum([len(x._data.axes[0]) for
                                           x in self.objs])]
             indexer = np.concatenate([offsets[i] + b.ref_locs
@@ -1052,12 +1053,21 @@ def _concat_blocks(self, blocks):
                 concat_items = indexer
             else:
                 concat_items = self.new_axes[0].take(indexer)
-
+                
             if self.ignore_index:
                 ref_items = self._get_fresh_axis()
                 return make_block(concat_values, concat_items, ref_items)
 
-            return make_block(concat_values, concat_items, self.new_axes[0])
+            block = make_block(concat_values, concat_items, self.new_axes[0])
+
+            # we need to set the ref_locs in this block so we have the mapping
+            # as we now have a non-unique index across dtypes, and we need to
+            # map the column location to the block location
+            # GH3602
+            if not self.new_axes[0].is_unique:
+                block._ref_locs = indexer
+
+            return block
 
     def _concat_single_item(self, objs, item):
         all_values = []


Test patch:
diff --git a/pandas/tools/tests/test_merge.py b/pandas/tools/tests/test_merge.py
--- a/pandas/tools/tests/test_merge.py
+++ b/pandas/tools/tests/test_merge.py
@@ -1682,6 +1682,20 @@ def test_concat_bug_2972(self):
         expected.columns=['same name', 'same name']
         assert_frame_equal(result, expected)
 
+    def test_concat_bug_3602(self):
+
+        # GH 3602, duplicate columns
+        df1 = DataFrame({'firmNo' : [0,0,0,0], 'stringvar' : ['rrr', 'rrr', 'rrr', 'rrr'], 'prc' : [6,6,6,6] })
+        df2 = DataFrame({'misc' : [1,2,3,4], 'prc' : [6,6,6,6], 'C' : [9,10,11,12]})
+        expected = DataFrame([[0,6,'rrr',9,1,6],
+                              [0,6,'rrr',10,2,6],
+                              [0,6,'rrr',11,3,6],
+                              [0,6,'rrr',12,4,6]])
+        expected.columns = ['firmNo','prc','stringvar','C','misc','prc']
+
+        result = concat([df1,df2],axis=1)
+        assert_frame_equal(result,expected)
+
     def test_concat_series_axis1_same_names_ignore_index(self):
         dates = date_range('01-Jan-2013', '01-Jan-2014', freq='MS')[0:-1]
         s1 = Series(randn(len(dates)), index=dates, name='value')