Python DataFrameの重複行の削除

### DataFrameの重複行の削除

`DataFarme`を結合して重複があった場合の削除方法


```python
import pandas as pd
df1 = pd.DataFrame([['2022/1/6', '3000', '2400', '240'],
                    ['2022/1/7', '1000', '800', '80'],
                    ['2022/1/8', '3000', '2400', '240'],
                    ['2022/1/9', '1500', '1200', '120']],
                   columns=['date', 'value', 'cost', 'tax'])
df1['date'] = pd.to_datetime(df1['date']).dt.date
df2 = pd.DataFrame([['2022/1/8', '3000', '2400', '240'],
                    ['2022/1/9', '1500', '1200', '120']],
                   columns=['date', 'value', 'cost', 'tax'])
df2['date'] = pd.to_datetime(df2['date']).dt.date
df3 = pd.DataFrame([['2022/1/9', '1500', '1200', '120', '15'],
                    ['2022/1/10', '500', '400', '40', '5'],
                    ['2022/1/11', '1000', '800', '80', '10']],
                   columns=['date', 'value', 'cost', 'tax', 'tax2'])
df3['date'] = pd.to_datetime(df3['date']).dt.date
```


```python
df = pd.concat([df1, df2, df3], ignore_index=True)
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2022-01-08</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>4</th>
      <td>2022-01-08</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>5</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>15</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>



index「2, 4」と「3, 5, 6」にdateの重複があります。dateの重複を判定したいので`subset='date'`を引数にします。
`subset`の引数がない場合は行のすべてが重複す場合に削除となります。
また、最初を残す(default)か、最後を残すかを選択できます。
最初を残す( `keep='first'`)と、index「2」と「3」が残ります。


```python
df.drop_duplicates(subset='date', keep='first')
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2022-01-08</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>



最後を残す( `keep='last'`)と、index「4」と「6」が残ります。


```python
df.drop_duplicates(subset='date', keep='last')
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>4</th>
      <td>2022-01-08</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>15</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>


複数のcolumnで判定した場合は`subset`に複数のcloumnをセットする


```python
df.drop_duplicates(subset=['date', 'tax2'])
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2022-01-08</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>6</th>
      <td>2022-01-09</td>
      <td>1500</td>
      <td>1200</td>
      <td>120</td>
      <td>15</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>


重複している行すべてを削除する場合は`keep=False`


```python
df.drop_duplicates(subset='date', keep=False)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>





indexの振り直しもできます。`ignore_index=True`


```python
df.drop_duplicates(subset='date', keep=False, ignore_index=True)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>3</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>



重複行を判定する場合は`duplicated`を使います。引数の`subset`、`keep`の使い方は同じです。


```python
df.duplicated(subset='date', keep=False)
```




    0    False
    1    False
    2     True
    3     True
    4     True
    5     True
    6     True
    7    False
    8    False
    dtype: bool



重複行を判定した結果を使っての削除


```python
df[~df.duplicated(subset='date', keep=False)]
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>date</th>
      <th>value</th>
      <th>cost</th>
      <th>tax</th>
      <th>tax2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2022-01-06</td>
      <td>3000</td>
      <td>2400</td>
      <td>240</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2022-01-07</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>7</th>
      <td>2022-01-10</td>
      <td>500</td>
      <td>400</td>
      <td>40</td>
      <td>5</td>
    </tr>
    <tr>
      <th>8</th>
      <td>2022-01-11</td>
      <td>1000</td>
      <td>800</td>
      <td>80</td>
      <td>10</td>
    </tr>
  </tbody>
</table>
</div>