Repository: electricitymaps/electricitymaps-contrib

Issue:
Switch Sweden data source to SVK instead of ENTSOE
- ENTSOE data is not good enough at the moment: Total production data is incorrect (Some major CHP plants are missing), and the "other" category does not break down fossil from biomass. 

- Sourcing ENTSOE inputs: At the moment, Svenska Kraftnät is the Swedish authory responsible for the reporting to ENTSOE, and depend on reports from another authority, Energimyndigheten, to update the information. They are aware of that their categories are not broken down enough, and are working on a solution to solve this. No information on when this project is due.

- Meanwhile, [SVK](http://www.svk.se/en/national-grid/the-control-room/) and [Stattnet](http://www.statnett.no/en/Market-and-operations/Data-from-the-power-system/Nordic-power-balance/) are providing better data: with less unknown and updated every 15 min

- Therefore, I suggest not to wait for ENTSOE data to get better, and start instead right now to

- [x] Write a new parser sourcing data from one of these two sites 

- [ ] Transform the "Thermal" category into (72% Biofuel, 11% coal, 8%unknown, 7%gas, 2%oil or (88% Biofuel, 10% other, 1%coal, 1%gas, 1%oil.) if industrial autoconsumption is not included in SVK data, as I suspect. These statistical breakdown comes from an analysis of thermal power production in the previous years, as found out after investigation from [energimyndigheten](https://www.energimyndigheten.se/en/facts-and-figures/publications/) and [Swedish Energy Agency](https://energimyndigheten.a-w2m.se/Home.mvc?ResourceId=5545)



Discussion:
Regarding the second checkbox, assuming breakdown of "unknown" or "thermal" by last year's breakdown has been suggested for quite a few other locations and always turned down so far ;)
Regarding the second checkbox, I would be against (as in the past) to distribute the unknown category across biofuel and others. The reason is that using a static distribution might be misleading. We also do want to highlight that the data is good enough, making as few assumptions as possible.
I do however think we could refine the GHG emission factor based on that breakdown, in order to avoid assuming a coal/gas mix in unknown.

Olivier
Make sense...So we won't have a better breakdown but a better GHG intensity, absolute prod value, and faster refresh rate. Still worth doing it to my view
Few clarifications/additions.

* "Svenska  Kraftnät" and "Statnett" are system operators (TSO) for Sweden and Norway respectively. 
* Svenska Kraftnät uses data from Satnett on their website.
* SvK and "Svenska Kraftnät" are the same. 
* "Energimyndigheten" and "Swedish Energy Agency" are the same too.
Some code. 

Flow:
`timestamp = unixtime;`
`urlflow = 'http://driftsdata.statnett.no/restapi/PhysicalFlowMap/GetFlow?Ticks=' + timestamp + '&rnd=' + Math.floor((Math.random() * 10000000) + 1);`

Note that, temporary Statnett returns wrong direction of EE->LV

Production:
`url= 'http://driftsdata.statnett.no/restapi/ProductionConsumption/GetLatestDetailedOverview?timestamp='+timestamp`
Good news, i've had confirmation that VSK/Stattnet data does not include industrial auto-consumption. We can safetly use  the static mix of **88% Biofuel, 10% other, 1%coal, 1%gas, 1%oil i**n the "Thermal" category. 
@brunolajoie you mean for the carbon intensity of "unknown" right?
Statnett has a category "thermal". The static mix that @brunolajoie refers to I assume is to translate to the categories used on electricity map. It is also safe to assume that Statnett category "unknown" is mostly thermal. The static mix that @brunolajoie mentions is for Sweden. The other Nordpool countries each have a different mix. I.e. Estonia mostly oil, Denmark and Finland quite a bit more coal.
@corradio I was talking about "Thermal" category on Stattnett/SVK for Sweeden.
Their "other" category is just totally unknown. We have no other choice than to use our "unknown" category.
We're mixing several things here. I will try to clarify.
1/ The mapping of the categories: We shouldn't break down the thermal from SVK and distribute to EM categories (biomass etc..) because that is a crude assumption. Furthermore, it's a static mapping. On the Electricity Map we want to see dynamic data as close to the source, without making any assumptions. I therefore suggest, as stated in https://github.com/tmrowco/electricitymap/issues/872#issuecomment-347440492, to map SVK thermal + SVK unknown to EM unknown.
2/ The carbon intensity of the categories: We should set a custom carbon intensity for the EM "unknown". We can do that based on the static breakdown that @brunolajoie refers to, assuming SVK unknown is the same as SVK thermal?
1/ Clear

2/ Therefore, we need to change my above-mentioned ratio, by considering SVK "unspecified" to be a 700gCO2/kWh default thermal fuel. Here is the final result 

![image](https://user-images.githubusercontent.com/22095643/33476546-a725e934-d682-11e7-9c55-d6b2b94ea959.png)

I just checked your assumptions. In 2016 the emissions from both heat and power was 4.781 million tons of CO2 according to official statistics from [Naturvårdsverket](http://www.naturvardsverket.se/Sa-mar-miljon/Statistik-A-O/Vaxthusgaser-utslapp-fran-el-och-fjarrvarme/). The total production from _Unspecified_ and _Thermal_ categories from Statnett were 12.2 TWh in 2016 according to my own database for [elstatstik.se](http://elstatistik.se).

Emissions from only power production are not specified as most of power production in Sweden from thermal sources is from co-generation, combined heat and power so this makes sense.

**The emissions per kWh were thus equal or less then 4.781 / 12.2 = .392 kg per kWh.** 

@brunolajoie assumption of 362 gram per kWh for combined _Thermal_ and _Unspecified_ is thus perfect imo. 
Conclusion. I agree with @corradio to combine Statnett categories _Unspecified_ and _Thermal_ into the electricity map category _Unknown_ and use @brunolajoie assumption of 362 gCO2eq/kWh.
Sweden has four regions with different prices for electricity. If we are to present the price correctly, Sweden must be divided into these regions. Is this something considered?
 See [Electricity sector in Sweden](https://en.wikipedia.org/wiki/Electricity_sector_in_Sweden#Transmission,_import_and_export)
@magol yes indeed. The problem is that ENTSOE does not report production per bidding zone for now..

Resolving patch:
diff --git a/mockserver/server.py b/mockserver/server.py
new file mode 100644
--- /dev/null
+++ b/mockserver/server.py
@@ -0,0 +1,18 @@
+#! /usr/bin/env python
+
+# Usage: python __file__.py <port>
+
+from SimpleHTTPServer import SimpleHTTPRequestHandler, BaseHTTPServer
+
+class CORSRequestHandler(SimpleHTTPRequestHandler):
+    def do_OPTIONS(self):
+        self.send_response(200, 'OK')
+        self.end_headers()
+
+    def end_headers(self):
+        self.send_header('Access-Control-Allow-Origin', '*')
+        self.send_header('Access-Control-Allow-Headers', 'x-request-timestamp, x-signature, electricitymap-token')
+        SimpleHTTPRequestHandler.end_headers(self)
+
+if __name__ == '__main__':
+    BaseHTTPServer.test(CORSRequestHandler, BaseHTTPServer.HTTPServer)
diff --git a/parsers/statnett.py b/parsers/statnett.py
new file mode 100644
--- /dev/null
+++ b/parsers/statnett.py
@@ -0,0 +1,77 @@
+# The arrow library is used to handle datetimes
+import arrow
+# The request library is used to fetch content through HTTP
+import requests
+
+
+def fetch_production(country_code='SE', session=None):
+    """Requests the last known production mix (in MW) of a given country
+
+    Arguments:
+    country_code (optional) -- used in case a parser is able to fetch multiple countries
+    session (optional)      -- request session passed in order to re-use an existing session
+
+    Return:
+    A dictionary in the form:
+    {
+      'countryCode': 'FR',
+      'datetime': '2017-01-01T00:00:00Z',
+      'production': {
+          'biomass': 0.0,
+          'coal': 0.0,
+          'gas': 0.0,
+          'hydro': 0.0,
+          'nuclear': null,
+          'oil': 0.0,
+          'solar': 0.0,
+          'wind': 0.0,
+          'geothermal': 0.0,
+          'unknown': 0.0
+      },
+      'storage': {
+          'hydro': -10.0,
+      },
+      'source': 'mysource.com'
+    }
+    """
+    r = session or requests.session()
+    timestamp = arrow.now().timestamp * 1000
+    url = 'http://driftsdata.statnett.no/restapi/ProductionConsumption/GetLatestDetailedOverview?timestamp=%d' % timestamp
+    response = r.get(url)
+    obj = response.json()
+
+    data = {
+        'countryCode': country_code,
+        'production': {
+          'nuclear': float(filter(
+            lambda x: x['titleTranslationId'] == 'ProductionConsumption.%s%sDesc' % ('Nuclear', country_code),
+            obj['NuclearData'])[0]['value'].replace(u'\xa0', '')),
+          'hydro': float(filter(
+            lambda x: x['titleTranslationId'] == 'ProductionConsumption.%s%sDesc' % ('Hydro', country_code),
+            obj['HydroData'])[0]['value'].replace(u'\xa0', '')),
+          'wind': float(filter(
+            lambda x: x['titleTranslationId'] == 'ProductionConsumption.%s%sDesc' % ('Wind', country_code),
+            obj['WindData'])[0]['value'].replace(u'\xa0', '')),
+          'unknown':
+            float(filter(
+              lambda x: x['titleTranslationId'] == 'ProductionConsumption.%s%sDesc' % ('Thermal', country_code),
+              obj['ThermalData'])[0]['value'].replace(u'\xa0', '')) +
+            float(filter(
+              lambda x: x['titleTranslationId'] == 'ProductionConsumption.%s%sDesc' % ('NotSpecified', country_code),
+              obj['NotSpecifiedData'])[0]['value'].replace(u'\xa0', '')),
+        },
+        'storage': {},
+        'source': 'driftsdata.stattnet.no',
+    }
+
+    # Parse the datetime and return a python datetime object
+    data['datetime'] = arrow.get(obj['MeasuredAt'] / 1000).datetime
+
+    return data
+
+
+if __name__ == '__main__':
+    """Main method, never used by the Electricity Map backend, but handy for testing."""
+
+    print 'fetch_production(SE) ->'
+    print fetch_production('SE')


Test patch: