Repository: electricitymaps/electricitymaps-contrib

Issue:
Taiwan real-time data has stopped working
Taiwain seems to have been offline recently
It used to work correctly, something may have changed in the data source?

Kibana error description [here](https://kibana.electricitymap.org/app/kibana#/discover/10af54f0-0c4a-11e9-85c1-1d63df8c862c?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:'@timestamp',negate:!f,params:(query:'2019-02-13T09:56:26.971Z',type:phrase),type:phrase,value:'February%2013th%202019,%2010:56:26.971'),query:(match:('@timestamp':(query:'2019-02-13T09:56:26.971Z',type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',asc)))



Discussion:
This is the error that is being raised.
```shell
Traceback (most recent call last):
  File "/home/feeder/lib/fetch_data.py", line 181, in launch_parsers
    objs = res.get(timeout)
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/feeder/lib/fetch_data.py", line 179, in <lambda>
    **parser_kwargs))
  File "/home/contrib/parsers/TW.py", line 28, in fetch_production
    objData['fueltype'] = objData.fueltype.str.split(')').str[0]
  File "/usr/local/lib/python3.6/site-packages/pandas/core/generic.py", line 3610, in __getattr__
    return object.__getattribute__(self, name)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/accessor.py", line 54, in __get__
    return self.construct_accessor(instance)
  File "/usr/local/lib/python3.6/site-packages/pandas/core/strings.py", line 1910, in _make_accessor
    raise AttributeError("Can only use .str accessor with string "
AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas
```

edit: I think the error is due to some changes in pandas but I need to understand what exactly the parser is doing first.
Looking at the listed [source](http://www.taipower.com.tw/d006/loadGraph/loadGraph/genshx_.html) in the readme it seems the data is supplied from http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/genary.txt, which is different to the source in the parser (http://data.taipower.com.tw/opendata01/apply/file/d006001/001.txt)

edit swapping the url makes the parser run successfully.

```shell
(EM-env) chris@ThinkPad:~/electricitymap$ python parsers/TW.py 
parsers/TW.py:34: FutureWarning: convert_objects is deprecated.  To re-infer data dtypes for object columns, use DataFrame.infer_objects()
For all other conversions use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.
  objData = objData.convert_objects(convert_numeric=True)
{'zoneKey': 'TW', 'source': 'taipower.com.tw', 'datetime': datetime.datetime(2019, 2, 14, 22, 10, tzinfo=tzfile('/usr/share/zoneinfo/Asia/Taipei')), 'storage': {'hydro': -800.6999999999999}, 'production': {'hydro': 338.90000000000003, 'oil': 512.5, 'gas': 9280.2, 'unknown': 1435.8, 'solar': 0.0, 'wind': 191.20000000000002, 'coal': 8708.6, 'nuclear': 3845.4}}
```
 
Good catch! Let us know if you see it as quickly fixeable or if it need a entire rewrite
Actually looking closely at http://www.taipower.com.tw/tc/page.aspx?mid=206&cid=404&cchk=8ccc1918-8cae-4f40-a2d0-b43454f4f218, there's a request to http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/loadfueltype.csv for the underlying graph data.

That csv contains data for the entire day so it could provide us with way to return a list of datapoints rather than just one.

So we have a choice, either swap the url and fix the pandas warning or switch to using that csv.
Oh cool, swapping the URL to http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/genary.txt indeed fix the problem easily. I'll do it right now to stop the bleeding.

Now, to chose between http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/genary.txt  and http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/loadfueltype.csv, a good argument could be if one could fetch historical data. Else, I'm not sure it's usefull to spend time coding a new parser

Resolving patch:
diff --git a/parsers/TW.py b/parsers/TW.py
--- a/parsers/TW.py
+++ b/parsers/TW.py
@@ -9,7 +9,7 @@ def fetch_production(zone_key='TW', session=None, target_datetime=None, logger=N
     if target_datetime:
         raise NotImplementedError('This parser is not yet able to parse past dates')
 
-    url = 'http://data.taipower.com.tw/opendata01/apply/file/d006001/001.txt'
+    url = 'http://www.taipower.com.tw/d006/loadGraph/loadGraph/data/genary.txt'
     response = requests.get(url)
     data = response.json()
 


Test patch: