Best hour of the day - Page 1
Forum section: electronics / general

author: paulca
author_rank: Super Contributor
timestamp: March 06, 2023, 05:13:23 pm
content: I have about 6 months of solar data. The data is about to change as I upgraded the panel.Before it has much effect, I wanted to ask the data a simple enough question. "What hour of the day do I receive the most solar power".I store the data in InfluxDB, which is (alegedly) an enterprise grade time series database. But, it lacks even the most basic of time and date functions outside of "groupby time". Enough of a rant about it. I exported the data and employed the big guns.Code:docker-compose exec influxdb influx --database home_auto --execute "select * from one_year.solar" --format json > ../../../spark/compose1/data/solar_data.jsonThis produces a 60Mb JSON file. Which has the 5 minute average, max and min solar values for the past 6 months or so.Quickly grab a "spark cluster in a can":Code:curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/spark/docker-compose.ymlAbracadabra up some PySpark analysis code:Code:import pysparkfrom pyspark.sql import SparkSessionfrom pyspark.sql.functions import *spark = SparkSession.builder.appName('barney').master("spark://spark:7077").getOrCreate()df = spark.read.json("/var/data/solar_data.json")df = df.select(explode("results.series"))df = df.select(explode("col"))df = df.select("col.*")df = df.select(col("name"), explode("values"))df = df.select(col("col").alias("time"),col("col").alias("key"),col("col").alias("maxVal"),col("col").alias("minVal"),col("col").alias("name"),col("col").alias("units"),col("col").alias("value"))df = df.withColumn("timestamp", from_unixtime(df.time / 1000 ** 3,"yyyy-MM-dd HH:mm:ss:SSS"))df = df.withColumn("timestamp2", to_timestamp("timestamp", "yyyy-MM-dd HH:mm:ss:SSS"))df = df.withColumn("hod", hour(df.timestamp2))df = df.select(col("hod"), col("maxVal").cast('numeric').alias("value_dbl"))stats = df.filter("key='panelPower'").groupBy("hod").mean("value_dbl").alias("Mean Power").orderBy("hod")stats.show(25, False)And... run it...Code:docker-compose exec spark spark-submit /var/data/best_hour.pyAnd I get the output:Code:+---+--------------+|hod|avg(value_dbl)|+---+--------------+|0 |0.0000 ||1 |0.0000 ||2 |0.0000 ||3 |0.0000 ||4 |0.0338 ||5 |0.2353 ||6 |0.5874 ||7 |1.2419 ||8 |2.2043 ||9 |4.2080 ||10 |6.4192 ||11 |9.6325 ||12 |10.3250 ||13 |7.7229 ||14 |3.4613 ||15 |2.1370 ||16 |1.3422 ||17 |0.8659 ||18 |0.4847 ||19 |0.2003 ||20 |0.0288 ||21 |0.0000 ||22 |0.0000 ||23 |0.0000 |+---+--------------+Now, I hear you, I hear you... That's an awfully long way to go about finding out that "Noon" is the best time of day for solar.Those hours are UTC/GMT. I am at 8* west. My solar maxima is around 12:40 GMT.The panel is shaded in the mornings by the house and the hedge in the evenings. Late morning through early evening is unshaded. This data, assuming it remains consistent by the time I schedule the panel fitting to the roof, allows me to bias the panel towards the west side of the roof and capture slight more early morning and slightly less evening sun.

author: Jeroen3
author_rank: Super Contributor
timestamp: March 06, 2023, 06:41:35 pm
content: I did a calculation like this with the sqlite data from my p1monitor (data logger on energy meter) to determine how much the projected PV output from my WSW faced system would be covering my own consumption during a year. Eg: if it was viable without net-metering.Turns out it was, it was able to cover about 30% of my own consumption. evidenced by 2022 my pv system made 2500 kWh, I exported 1700.This put the ROI waaay over at 8 years or so. However...withnet-metering (I had net 16 kwh last year) and current world chaos energy prices it has paid for itself already, only 2.5 years.Projected PV input can be calculated very easily via sites like this:https://www.thomasberger.be/pv/pvcalc/So no need to measure it at all. Earth and sun orbits are very predicatable.My peak hours are around 14:00-16:00 depending on the time of the year.

author: paulca
author_rank: Super Contributor
timestamp: March 07, 2023, 07:42:42 pm
content: I should have mentioned the exercise was two fold, running spark locally at home, and processing influxdb data.The real challenges I'm hoping to develop into here are answer some bigger questions.Top of the list are:* Frequency/window analysis of heating requests against temperature delta.* Delta grouping of power usage - this then yields detection of individual devices when monitoring a common power feed - if you can get it to work.The former is tricky as the heating requests/demands are "single ended" fire and forget, refresh to timeout. So they only exist in the "ON" form and are only logged in the "ON" form. So scarring through a years 5 min data trying to work out how long the heating has been on, provided the "ON" events and the "timeout" value. Then present with the temp delta for outdoor vs. coldest/warmest room. etc.The later is more research. Some companies that do this use "artifical learning", which is just bullshit for remembering stuff.That is what spark (and many others) can do. Sure you can do that above in SQL too.The reason I'm using spark is simple cause I have "them'z skillz" from work. It's not a case of "I have a hammer, what looks like a nail", it's more, "Ah Ha! Finally a nail in hobby space I get to hit with the proverbial sledge hammer"