Smart light switch

= Legal disclaimer =
Do not attempt to repeat this project if you do not have a significant experience dealing with high voltage (AC) circuits! AC (mains; 120V or 220V depending on the country) electricity is deadly! If you are not comfortable dealing with AC, ask a qualified electrician for help, otherwise don't do it! I shall bare no responsibility if you'll get hurt or die while trying to follow my description, or if there will be any property damage (improperly done AC wiring can cause fire). This modification can void your home insurance - check with you insurer first to be sure.
To put it simply: if you are not an experienced electrician, don't do it! Consider instead the much safer option based on a servo (see below).
= Introduction =
I wanted to modify the two exterior light switches in my house (front, with three bulbs, and backyard), so it would be
# automatically turning the lights on when it gets dark outside, and off when it gets bright, with a bit of randomness,
# safe and reliable (so one could leave it on while traveling),
# cheap,
# totally non-obvious (both in terms of the appearance and behaviour),
# and both locally (with the physical switch) and remote (over the internet) controllable.
Using bulb adapters with a photosensor was a no-go, as it would violate my requirements #4 and #5. (As I have three bulbs in the front, located far from each other, they will definitely be turning on and off at different times, which will be very obvious; also, these adapters are very obvious in appearance.) I suspect the new WiFi controllable LED light bulbs will also have the synchronization issue, plus they are rather expensive.
As I already knew how to program and use Arduino and ESP8266 microcontrollers, in particular with AC solid state relays (from my 3D Guardian project - environment monitor for 3D printers), I went ahead and designed and built my own smart switch, which I believe achieved all my design goals. It is based on the popular ESP8266 chip with built-in WiFi (specifically the "NodeMCU devkit v0.9 CH340G" module which can be found on ebay for under $4), with the total costs ~7$ in parts. 
It is totally non-obvious - no visible changes in the appearance of both the lights and the switch. In particular, it doesn't use a photosensor (which would be quite obvious) to detect the dark time of the day; instead, it relies on Internet time (NTP) plus internal timer, and a built-in sunset/sunrise calculator to achieve this goal.
NEW: now it can also operate a mechanical light switch via a servo (like 9G). Here is a short youtube demo.
= Features = 
* It can be used for exterior and interior lights, with and without a physical switch.
* It uses the light switch already available for turning on/off the light. It can only be used with a single group switch. (So cannot be used in situations when the same lights can be turned on/off with more than one switch.)
* In the current design it can only be used with up to 60W (may be 100W) actual load, ideally LED bulbs. (For example it's totally fine for my 3 LED bulbs which are 100 + 60 + 60 = 220 "equivalent Watt".)
* It doesn't change the appearance of your light switch - it will look exactly as before outside.
* It has two main modes - "dumb" and "smart".
* When powered up, initially it is in the "dumb" mode - you can use the physical switch to turn the light on/off.
* Once it connects to your WiFi network, and if an internet connection is available, it uses NTP internet service to detect the current date and time, and then automatically switches to "smart" mode. After that it is no longer critical to have the WiFi/internet connection.
* Once in the smart mode, it computes the sunset and sunrise times for your location, plus some random deviation (say, between -5 ...+5 minutes) to turn on the lights around/soon after sunset time, and turn them off right before/around the sunrise time. One can fine-tune how dark it should get for the lights to be on, by changing the Z_ANGLE parameter in config.h.
* Once it connects to WiFi, it will try to join your local home automation network (using MQTT protocol). You'll have to run an MQTT server (on your Windows PC / Raspberry Pi / etc) for this to work. Also you'll need OpenHab server to be running (on the same computer) for full functionality. Once you have the MQTT and OpenHab servers running, you can add many more smart devices to your home smart network, including multiple instances of this smart light switch.
* Once it is connected to WiFi, an internal blue LED will optionally turn on. (It is quite visible through the switch cover.) This is disabled by default. 
* Once it is connected to the MQTT network, you can use a browser on your smart phone or PC to check the status of the light, and to control it remotely. If properly set up (using the Apache web server with the reverse proxy feature and SSL encoding), this can be securely accessed anywhere in the world.
* At any point the gadget can be switched between the dumb (fully manual) and smart modes - either remotely from a browser, or using the physical switch. In the latter case, one needs to flip the physical switch on-off three times within 4 seconds to toggle the mode (dumb->smart or smart-> dumb).
* If at any point WiFi and/or MQTT connections are lost, the switch will continue its operation normally using the current mode (smart or dumb). One can still control it using the physical switch. It will regularly try to reconnect to WiFi and/or MQTT until it succeeds.
* In the dumb mode, the lights can be turned on and off either using the physical switch, or remotely from a browser.
* Regardless of where the mode (smart/dumb) or the lights (on/off) are switched - remotely or with the physical switch - you'll get an instant notification of the new state in the browser.
* Accessing from a browser also gives you some extra information: the time of the next flip and the time left until the next flip (only in smart mode), the current position of the physical switch (on/off), and the current/historically maximum temperatures of the solid state relay used to switch the lights.
* You can always switch the lights on or off remotely, regardless of the current position of the physical switch. The same is true in the opposite direction: regardless of the current online settings, one can always use the physical switch to switch between smart<->dumb modes or turn the lights on/off (only in dumb mode).
* If the solid state relay starts failing and will get overheated (>60C), the lights will be turned off, and you'll get a visual warning (internal LED will flash fast). Also, the Alarm message will be displayed in the browser. If that happens, you should look inside the switch to see what's the problem (but don't forget to disable the AC voltage first from your house's switchboard!), and only if satisfied you can power cycle the controller to revert it back to normal operation.
* If everything else fails, the thermal fuse will power down the solid state relay and the lights once the relay gets to 130C.
* The gadget has a built-in protection against abuse. E.g. if the physical switch is turned on/off (either in reality, or because of the fake readings caused by the electrical interference from AC wiring) more than 100 times per hour, it will be disabled (one can still operate it online), the gadget will switch to smart mode (if the time is known), or turn off the lights (if the time is not known). An internal LED will be flashing slowly as a warning. The only way to revert back to normal operation is by power cycling the switch (from the home's switchboard with breakers).
* Another protection is against hackers: if the online switches (smart<->dumb and/or on/off) are used > 100 times per hour, these switches are disabled (one can still operate the gadget using the physical switch). The rest is the same as in the previous point (slowly flashing warning LED; requires power cycling to revert to normal operation).
* In my design it is fairly easy to update the firmware in the switch: just shut down the power to the switch from the house's switchboard, remove the plastic cover of the switch - this will expose the micro-USB connector of the controller, connect your laptop to the controller using a USB cable, and upgrade the firmware using the Arduino IDE. Never enable AC power to the switch when the laptop is connected to the controller, or more generally - when the light's plastic cover is off!
* If you ever need to reboot the switch, just use the corresponding breaker (turn it off then on) in your house's switchboard.
Don't worry - despite the long list of the features, this is designed to be used as "turn it on and forget": most likely you'll never need to touch the physical switch, the smart switch will be doing its thing by default, and you will only need (or want) to use the online access to see if the light is on or off.
Some additional, more technical points:
* The code uses programmatic debouncing when reading the state of the physical switch. (This makes it much more reliable.)
* Some information is permanently stored in the flash memory, and some of it can only be accessed when recompiling the code in DEBUG mode. These are: the historically largest SSR temperature recorded (online and DEBUG), the corresponding date and time (only DEBUG), and the total number of reboots (only DEBUG).
* The device works much better with a small cheap 5V PSU (providing voltage to the +5V pin of the ESP8266 board), compared to a small cheap 3.3V PSU connected to the +3.3V pin of ESP8266 board. In particular, analogue pin readings (temperature measurement) become very noisy (unusable) with the 3.3V scenario. I suspect this is due to the stabilizing effect of the 5V-3.3V voltage converter built into the devkit.
* One can add a small (0.1A / 250V) fuse before the PSU.
Now the smart switch can also operate a mechanical light switch via a servo (like 9G). This is the safest way to use my smart switch, as no electric work inside the wall is needed, and the fire hazard is zero. But it's also the least convenient approach, as the mechanical switch is no longer usable (as long as the servo mechanism is attached to the switch), so only Smart and remote control functionality is available. A short youtube demo.
= The parts used =
* ESP8266 controller (NodeMCU devkit v0.9 CH340G): ~4$ on ebay (e.g. http://www.ebay.ca/itm/272526162060 )
* AC solid state relay (250V / 2A): 1$ (http://www.ebay.ca/itm/282227077259)
* Tiny AC-DC adapter (5V / 0.7A): 1.25$ (http://www.ebay.ca/itm/152312035099)
* Two resistors (47k and 3.3k) and a 50k thermistor: a few cents
* Thermal fuse (250V / 8A / 130C): I used one I found in a broken digital projector - this is a multiple use fuse. Or one can use a single-use fuse, like this one: 250V / 3A / 115C, 1.80$ for 10 pcs (http://www.ebay.ca/itm/182390852074).
If you build a servo-operated mechanical smart switch, you only need the ESP8266 controller from the above list. In addition, you'll need a cheap 9G servo from ebay (e.g. https://www.ebay.ca/itm/112664443240), and any 1A USB power adaptor.
= Schematic = 
Be extremely careful with AC circuits - it is deadly! (Also see my Legal Disclaimer at the top of this page.) Always disable the AC power (using the corresponding breaker in the house's switchboard) to the AC circuit you will be working with, and always double/triple check it is indeed disabled (using AC voltage detectors etc) before touching any naked AC wire!
I used epoxy glue to glue both the thermistor and the thermal fuse to one surface of the SSR, then wrapped it around with Kapton tape (for electrical insulation - in case things start to melt down), and then used a metal bracket to screw it to the metal box containing the switch, inside the box.
Make sure it is the "hot" AC wire (normally black), and not the "neutral" wire (normally white), is the one that goes through the thermal fuse and the SSR to the lights.
You will need both the hot and neutral wires to be available inside the light switch box for this to work (the PSU will need both). This was the case with my light switch, and I suspect with most others.
For the controller and the PSU, I printed small enclosures using my 3D printer, and then just inserted them inside the switch metal box, without securing them to anything. (WiFi works surprisingly well, given that the controller is inside a metal box with only one - front - face non-metal.)
It is critical to use the 3.3k pullup resistor (R3) for the physical switch readout. The internal pullup resistor is way too weak (~100k), which in my experience results in huge rate (a few times per second!) of false switch readings due to the interference noise from the nearby AC wiring. Since I soldered the 3.3k external pullup resistor I've been getting zero false readings.
I like to use AC solid state relays (versus regular relays) - if the thermal/electrical conditions are right, they will serve forever (unlike regular relays). As a drawback, the SSR I'm using restricts me to use lights up to 60W (actual; perhaps up to 100W). Ideally, these should be LED lights, as they are the only ones which do not generate a significant inrush current (which might eventually kill your SSR) when you turn them on. Both fluorescent and incandescent bulbs are really bad in this regard - the initial current spike can be as high as ~10x the average value. BTW this is bad for both SSRs and regular relays.
But if you really want to control a higher wattage load, you can replace the SSR in my circuit with a regular relay with the specs you need (voltage and current). Thermistor in this setup should be placed near the controller or the PSU, as you can't really switch off a regular relay once it starts melting down. The thermal fuse should still be attached to the relay.
The option of a portable switch without a physical switch:
For the servo option I designed the following 3D printed parts (designs are available on thingiverse):
= The code = 
The ESP code is written using C++ with Arduino IDE. Both the ESP code and my OpenHab scripts can be obtained on github:
https://github.com/pulsar123/Smart-light-switch
Read the instructions in the header of the Lights.ino file to find out how to customize the code. Most importantly, you'll need to create a new file, private.h, which will contain your private information (WiFi SSID and password, geographical coordinates and the time zone, and the summer time rules).
For the very first run, uncomment the "#define INITIALIZE" line in config.h and upload the code. This will initialize the flash memory. After that comment the line out and re-upload the code, for normal operation.
It is a good idea to first test the controller (don't connect it to AC yet!) using the DEBUG mode (uncomment "#define DEBUG" in config.h), with the laptop attached to the controller via USB cable, and running the Serial Monitor inside the Arduino IDE. This will tell you if there are any issues connecting to WiFi, MQTT, NTP, reading the physical switch etc.
= MQTT, OpenHab, and Apache servers = 
If you want to use the remote control feature of the switch, you'll need to run on a computer (Windows, Linux PC or Raspberry Pi) inside your home network at least two servers: MQTT and OpenHab. (You can replace OpenHab with any other similar home automation server with MQTT support). Ina addition, if you want a secure internet access to the switch, you can run a web server (e.g. Apache) on the same computer, which will use a reverse proxy feature to make the access to OpenHab server secure. (Currently OpenHab2 does support SSL, but it doesn't have the authentication feature yet, so strongly not recommended to be exposed to the internet).
I already have a Windows 7 PC which is on 24/7, so I installed all these servers there, and only provide the instructions for a Windows PC. But it should be fairly straightforward to do the same on a Raspberry Pi computer.
== MQTT ==
The popular choice for a MQTT server is Mosquitto. I installed it as a Windows service using these instructions.
If you get the "Missing VCRUNTIME140.dll" error, you'll need to install https://www.microsoft.com/en-us/download/details.aspx?id=48145 .
== OpenHab ==
This is a popular home automation software. The current version is 2. It can be obtained here. I used NSSM program to run OpenHab as a Windows service. I added the MQTT binding inside the OpenHab, and used the following configuration for conf/services/mqtt.cfg file:
 broker.url=tcp://localhost:1883
 broker.clientId=openhab
 broker.retain=true
 broker.async=true
The example items, sites, and map files can be found in my github repository (inside the openhab folder), and should be copied to the "items", "sites", and "transform" subdirectories, respectively, inside "conf".
Then one can access the smart switch using the Basic UI interface as your_ip:8080 in your browser. This is totally insecure though, so don't do port forwarding to access OpenHab remotely. Instead, you should place it behind a reverse proxy using Apache (or NGinx) web server - see below.
A warning: if you make any modifications to the OpenHab files, you'll need to restart the OpenHab service, otherwise it'll be broken.
Since v0.2, my smart switch listens for mqtt topic "openhab/start", and if it receives the message "1" it will republish ("refresh") its status over mqtt. To make use of this feature, one has to modify the OpenHab launch script (forcing it to send "1" to "openhab/start" every time it restarts). For Windows, one has to add one extra line (starting with "start") in the start.bat file inside the OpenHab folder:
 @echo off
 
 echo Launching the openHAB runtime...
 
 setlocal
 set DIRNAME=%~dp0%
 start /b C:\openHAB2\mqtt_start.bat >nul
 "%DIRNAME%runtime\bin\karaf.bat" %*
Also, one needs to create the file mqtt_start.bat in the same folder with the following content:
 timeout /t 20 /nobreak
 C:\mosquitto\mosquitto_pub.exe -h 127.0.0.1 -t "openhab/start" -m "1"
 C:\mosquitto\mosquitto_pub.exe -h 127.0.0.1 -t "openhab/start" -m "0"
You should modify the paths accordingly for your setup (openhab and mosquitto folders).
Every time OpenHab service is restarted, the messages "1" and "0" will be sent to the mqtt topic "openhab/start" 20 seconds after the restart. This will force all smart switches in the house to refresh their status. One can also send this message manually if a manual refresh is needed. I added a button "Refresh all" in my openhab/lights.sites file for this purpose.
Here is my interface for the smart switch using OpenHab:
* Smart mode:
* Dumb mode:
== Apache ==
I used the open source web server Apache to make the access to OpenHab secure - using SSL encoding and authentication. The Apache server can be run as a Windows service
These are the changes I made to the conf/httpd.conf file:
* I enabled the following modules (removed "#" in front of the corresponding LoadModule line): mod_headers, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_hcheck, mod_proxy_html, mod_proxy_http, mod_rewrite, mod_slotmem_shm, mod_socache_shmcb, mod_ssl, mod_watchdog.
* I entered the appropriate values for ServerName and DocumentRoot. (The latter is not really needed for OpenHab, only for other stuff you want to put on the web site.)
* I uncommented the line
 Include conf/extra/httpd-ssl.conf
And then the additions to conf/extra/httpd-ssl.conf :
NameVirtualHost *:80
<VirtualHost *:80>
 ServerName your_domain
 Redirect permanent / https://your_domain
</VirtualHost>
<VirtualHost _default_:443>
 ServerAdmin admin@commaster.net
 ServerName your_domain
 SSLEngine On
 SSLCertificateFile "C:/Users/*.pem"
 SSLCertificateKeyFile "C:/Users/*.pem"
 SSLCertificateChainFile "C:/Users/*.pem"
 ProxyPass /public !
 ProxyPass /private !
 ProxyPass /.well-known !
 ProxyPass / http://localhost:8080/ retry=0 timeout=3600
 ProxyPassReverse / http://localhost:8080/
 <Directory "C:/WWW/public">
 AuthType None
 Require all granted
 </Directory>
 <Directory "C:/WWW/.well-known">
 AuthType None
 Require all granted
 </Directory>
 <Directory "C:/WWW/private">
 AuthType Basic
 AuthName "Restricted Content"
 AuthUserFile "C:/xxxx"
 Require valid-user
 </Directory>
 <Proxy *>
 RequestHeader set X-Forwarded-Proto "https"
 AuthType Basic
 AuthName "Authentication"
 AuthUserFile "c:/xxx"
 Require valid-user
 </Proxy>
</VirtualHost>
Here it is assumed that you own a domain name, your_domain (could be DDNS - noip.com etc), and that in addition to providing the reverse proxy functionality to the OpenHab server, you also want to share files in two locations - C:/WWW/public (public files), and C:/WWW/private (private files, password protected). The OpenHab is accessed from the root of your_domain, and requires authorization. 
All three services are encoded with SSL. I used the free service Let's Encrypt to generate the SSL certificate (the *.pem files). (The "C:/WWW/.well-known" section in the conf file is needed for the certification to work correctly.) The /private and / (OpenHab) locations are password-protected using the Apache's utility htpasswd.exe.