Issue: TypeError: unhashable type: 'list' when using playhouse.dataset with Postgres
Author: mivade
State: closed
Created: 2016-03-30 11:44:16

I'm attempting to use the `playhouse.dataset` to lazily write data to a Postgres database. The data looks like this:

``` python
{'beat1_lb_mon': 10.0,
 'beat1_lb_status': 0,
 'beat1_signal': 0.750122,
 'beat1_signalaux': 0.377197,
 'beat1_waveplate': 32830.0,
 'beat2_lb_mon': 10.0,
 'beat2_lb_status': 0,
 'beat2_signal': 0.388794,
 'beat2_signalaux': 0.238648,
 'counter0_freq': 20000375.73899996,
 'counter0_freqref': 5000000.0,
 'counter1_freq': 20000000.09199998,
 'counter1_freqref': 20000000.0,
 'counter2_freq': 59999999.972,
 'counter2_signal': 0.7489009999999999,
 'counter3_freq': 19548567.59900001,
 'counter3_signal': 0.377808,
 'counter_channels': 4,
 'counter_gatetime': 1.0,
 'cw1_freq': -80000000.06399998,
 'cw2_freq': -39548567.69099999,
 'laser_mldet_ac': 1168.0,
 'laser_mldet_dc': 321.0,
 'lb1_mon': 6.230468,
 'lb1_status': 2,
 'lb2_mon': 0.546822,
 'lb2_status': 2,
 'lb3_mon': 3.881493,
 'lb3_status': 0,
 'lb4_mon': 10.0,
 'lb4_status': 0,
 'lb5_mon': 0.593604,
 'lb5_status': 0,
 'lb6_mon': 0.96824,
 'lb6_status': 0,
 'offset_freq': 20000000.09199998,
 'offset_freqref': 20000000.0,
 'offset_signal': 0.8038329999999999,
 'offset_stage': -14948.0,
 'reprate_freq': 250000093.93475,
 'reprate_freqref': 246250000.0,
 'reprate_stage': -211159.0,
 'syncro_adc_channel2': 0.002084,
 'system_latency': 0.514801025390625,
 'system_locked': True,
 'system_locked_for': 13972.88214206696,
 'system_plo_ok': True,
 'timestamp': 1459337054.158}
```

Using a SQLite database for testing, everything seems to work properly including guessing of datatypes. However, when I switch to Postgres, I get the following traceback:

``` python
Traceback (most recent call last):
  File "/home/iontrap/src/brush/env/bin/brush", line 9, in <module>
    load_entry_point('brush==1.0.0b2', 'console_scripts', 'brush')()
  File "/home/iontrap/src/brush/brush/cli.py", line 55, in main
    IOLoop.instance().run_sync(lambda: sweep.run())
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/tornado/ioloop.py", line 453, in run_sync
    return future_cell[0].result()
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/tornado/gen.py", line 282, in wrapper
    yielded = next(result)
  File "/home/iontrap/src/brush/brush/sweep.py", line 97, in run
    self.write(data)
  File "/home/iontrap/src/brush/brush/sweep.py", line 67, in write
    id = table.insert(**data)
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/playhouse/dataset.py", line 213, in insert
    return self.model_class.insert(**data).execute()
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 3213, in execute
    cursor = self._execute()
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 2627, in _execute
    sql, params = self.sql()
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 3181, in sql
    return self.compiler().generate_insert(self)
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 1885, in generate_insert
    meta.get_primary_key_fields(),
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 4299, in get_primary_key_fields
    for field_name in self.primary_key.field_names]
  File "/home/iontrap/src/brush/env/lib/python3.4/site-packages/peewee.py", line 4299, in <listcomp>
    for field_name in self.primary_key.field_names]
TypeError: unhashable type: 'list'
```

The `write` method is defined as:

``` python
    def write(self, data):
        table = self.db[options.sql_table]
        with self.db.transaction() as txn:
            id = table.insert(**data)
            if id == 1:
                for index in self.indices:
                    try:
                        table.create_index([index])
                    except OperationalError:  # index already exists
                        pass
```

This is using peewee 2.8.0 and a Postgres 9.3 server.


I'm a bit at a loss. Can you tell me what `table.model._meta.primary_key` returns? As well as `table._meta.fields`. Thanks!


Basically it looks like your table has a composite primary key, and when asked for `table.model._meta.primary_key.field_names`, it's returning a list of lists?


Hm, that could be, although I don't quite understand why. I'll post the requested information tomorrow when I again have access to the real data as in the original issue.

In the meantime, I have an offline testing version which only uses a subset of the keys in the dict above. On my own computer running Postgres 9.4, there are no problems. So I am not sure if that is somehow related to using fewer keys/columns or if there is a version-specific bug.


Is there an `id` column present in any of the data you're loading?


I'm tempted to think this is just a data issue, but until you can get me the output (and even better the actual row data) when it crashes that'd be great.


There is no explicit `id` column; only the columns given above in the form of keys exist. The table also does not exist prior to the `table.insert` call.


I was able to resolve this problem by dropping the table which had been created prior to trying to write (but was empty) and letting peewee handle the table creation itself. However, before doing this, I checked the output of `table.model_class._meta.primary_key` and `table.model_class._meta.fields`. Using `sqlite:///:memory:`, on the first run, I get:

```
primary key
<peewee.PrimaryKeyField object at 0x7f64787617b8>
fields
{'id': <peewee.PrimaryKeyField object at 0x7f64787617b8>}
```

and after the initial insert:

```
primary key
<peewee.PrimaryKeyField object at 0x7fa7322206a0>
fields
{'beat1_lb_mon': <peewee.FloatField object at 0x7fa7322140b8>,
 'beat1_lb_status': <peewee.IntegerField object at 0x7fa7322141d0>,
 'beat1_signal': <peewee.FloatField object at 0x7fa732220fd0>,
 'beat1_signalaux': <peewee.FloatField object at 0x7fa732220c18>,
 'beat1_waveplate': <peewee.FloatField object at 0x7fa732220cf8>,
 'beat2_lb_mon': <peewee.FloatField object at 0x7fa732214160>,
 'beat2_lb_status': <peewee.IntegerField object at 0x7fa7322207b8>,
 'beat2_signal': <peewee.FloatField object at 0x7fa732214748>,
 'beat2_signalaux': <peewee.FloatField object at 0x7fa7322143c8>,
 'counter0_freq': <peewee.FloatField object at 0x7fa732214198>,
 'counter0_freqref': <peewee.FloatField object at 0x7fa732220ef0>,
 'counter1_freq': <peewee.FloatField object at 0x7fa732220da0>,
 'counter1_freqref': <peewee.FloatField object at 0x7fa7322147b8>,
 'counter2_freq': <peewee.FloatField object at 0x7fa732220f28>,
 'counter2_signal': <peewee.FloatField object at 0x7fa732220e48>,
 'counter3_freq': <peewee.FloatField object at 0x7fa732214208>,
 'counter3_signal': <peewee.FloatField object at 0x7fa732214048>,
 'counter_channels': <peewee.IntegerField object at 0x7fa732220f60>,
 'counter_gatetime': <peewee.FloatField object at 0x7fa732220eb8>,
 'cw1_freq': <peewee.FloatField object at 0x7fa732220e10>,
 'cw2_freq': <peewee.FloatField object at 0x7fa732220828>,
 'id': <peewee.PrimaryKeyField object at 0x7fa7322206a0>,
 'laser_mldet_ac': <peewee.FloatField object at 0x7fa732214400>,
 'laser_mldet_dc': <peewee.FloatField object at 0x7fa732220b70>,
 'lb1_mon': <peewee.FloatField object at 0x7fa732214390>,
 'lb1_status': <peewee.IntegerField object at 0x7fa732220b00>,
 'lb2_mon': <peewee.FloatField object at 0x7fa732214278>,
 'lb2_status': <peewee.IntegerField object at 0x7fa732214358>,
 'lb3_mon': <peewee.FloatField object at 0x7fa732220ac8>,
 'lb3_status': <peewee.IntegerField object at 0x7fa732220dd8>,
 'lb4_mon': <peewee.FloatField object at 0x7fa732214320>,
 'lb4_status': <peewee.IntegerField object at 0x7fa732220ba8>,
 'lb5_mon': <peewee.FloatField object at 0x7fa732214080>,
 'lb5_status': <peewee.IntegerField object at 0x7fa732220c88>,
 'lb6_mon': <peewee.FloatField object at 0x7fa732214438>,
 'lb6_status': <peewee.IntegerField object at 0x7fa732220d68>,
 'offset_freq': <peewee.FloatField object at 0x7fa732220cc0>,
 'offset_freqref': <peewee.FloatField object at 0x7fa732220f98>,
 'offset_signal': <peewee.FloatField object at 0x7fa732220b38>,
 'offset_stage': <peewee.FloatField object at 0x7fa732220e80>,
 'reprate_freq': <peewee.FloatField object at 0x7fa732220c50>,
 'reprate_freqref': <peewee.FloatField object at 0x7fa732214128>,
 'reprate_stage': <peewee.FloatField object at 0x7fa732220748>,
 'syncro_adc_channel2': <peewee.FloatField object at 0x7fa732214240>,
 'system_latency': <peewee.FloatField object at 0x7fa732214780>,
 'system_locked': <peewee.IntegerField object at 0x7fa7322142b0>,
 'system_locked_for': <peewee.FloatField object at 0x7fa7322140f0>,
 'system_plo_ok': <peewee.IntegerField object at 0x7fa7322142e8>,
 'timestamp': <peewee.FloatField object at 0x7fa732220d30>}
```

With Postgres, prior to the insert (where it crashes):

```
primary key
<peewee.PrimaryKeyField object at 0x7f4afa8afb00>
fields
{'id': <peewee.PrimaryKeyField object at 0x7f4afa8afb00>}
```

Is there any other testing you can suggest to help figure out why this fails when the empty table exists?


No idea, other than that the table perhaps is missing a primary key?