Relying on relays for critical systems - Page 1
Forum section: electronics / projects

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 12:38:52 pm
content: Recently, I've developed an interest in poultry. Not just eating it, but raising them. If you ask me why, I won't have a good answer. It's a lot of work and you can't save money on your grocery bill.Anyway, I constructed an incubator controlled by Arduino. It is somewhat crude, but functional. We successfully hatched 10 chicks our first time out. Currently, the Arduino input comes from a temperature/humidity sensor. No other sensors are used. Output is a simple board I made consisting of power supply and relay for 120VAC output. Heat source is a 150W incandescent light bulb. The incubator was designed as a platform that could be improved as time would allow. We were in a hurry to hatch our first clutch, so no redundancy or alarms were designed into the system. The intent was to add a variety of features as time would permit (automatic egg turner, redundancy, fail-safe, humidity control, digital display, convert to Atmel MCU instead of Arduino etc). That's the future. At this point, its just a high tech thermostat. Aside from the heat source, the only other output from the Arduino is to the serial port connected to a laptop. Every 8 seconds, the temperature and humidity are printed on a line. Approximately every 2 minutes, the heat source will flip-flop and a brief report is sent to the serial port indicating a change in state (on/off). The same function in the code that calls the report function, calls the function to toggle the heat source. One can't happen without the other.Two days ago, our guinea fowl kindly contributed a dozen eggs and we decided to enlist the incubator for a second time. This morning, we woke up to an incubator temperature at 119 degrees F. 19 degrees higher than incubation temperature. I scrolled back to the last report. It indicated that the lamp was "Off". The heat source state is NOT determined empirically, which would arguably be better, but considering that a 3 week incubation successfully completed, there aren't likely any bugs in the code. Counting the lines of output, I determined that the temperature ramped up to 119 over a 100 minute period. Too long. Eggs are toast... so to speak.I turned off the incubator. 10 minutes later I turned it back on. It worked flawlessly. At this point, I can only think the relay stuck closed. It's a 3 amp PCB relay (NO). The anomaly does not seem to be repeatable. No visible clues are on the Arduino or pcb. All looks good.So, now I'm considering what preventions I should implement. An alarm is obvious. As the Arduino was "aware" of the temperature problem. Also, some kind of fail-safe is in order. I'm considering an independent MCU with a dedicated temperature/humidity sensor that only outputs to a main relay that shuts down power to the entire system anytime a problem is detected. Is this overkill? If I take this route, I'd also link the Arduino so the Arduino could send a shut down order as it would have more complex code detecting more errors.I'd like to know more about mechanical relay reliability. Is this kind of failure expected? The lack of redundancy and alarm has never set well with me. In fact an alarm was on my list of things to do in the next day or two. How would you design the system and what are your thoughts on using relays on critical systems?

author: SL4P
author_rank: Super Contributor
timestamp: April 30, 2014, 12:47:56 pm
content: one possible thought - do you have a counter in code that is being reset / overflowing?Or if you ae sure it's the relay, add another in series - to provide double-open protection. You could also use a second pair of contacts for feedback of the relay action.

author: N2IXK
author_rank: Frequent Contributor
timestamp: April 30, 2014, 12:55:08 pm
content: You are switching a 150W incandescent lamp load with a 3A rated relay? That might be the cause of your problem right there.The 150W rating is the power consumption of the lamp ONCE LIT. But a tungsten filament lamp has a MUCH lower resistance when cold, resulting in an inrush current spike of 10 or more times the normal operating current. This can cause relay contacts to weld together if not properly rated for a tungsten load.You need to use a much more robust relay than lamp wattage would indicate when switching incandescent lamps.I would do a destructive analysis of that relay. Open it up and examine the contact faces. I'm betting you will see severe pitting and erosion.

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 01:01:23 pm
content: Quote from: SL4P on April 30, 2014, 12:47:56 pmGood thought on the counter, but no. Since the serial output demonstrates the MCU is aware of the temperature AND the code was run to turn off the relay, either the relay didn't get the LOW pin instruction or the relay failed. Loose signal connection maybe?Thanks for the second-pair-for-contacts suggestion! That would be the quickest and easiest short term solution. Regardless, the additional redundancy is never a bad thing.

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 01:08:49 pm
content: Quote from: N2IXK on April 30, 2014, 12:55:08 pmI honestly didn't consider that. I just checked the cold resistance of the bulb. 7 ohms

author: N2IXK
author_rank: Frequent Contributor
timestamp: April 30, 2014, 01:30:27 pm
content: Quote from: chipwitch on April 30, 2014, 01:08:49 pmThat would do it. Assuming the relay contacts close at the peak of the AC cycle, you are looking at an inrush current of ~25 amps. A little 3A relay isn't going to appreciate that very much.When shopping for a replacement, look for a relay with contacts that specifically carry a "T" rating for tungsten loads. You might also consider an inrush current limiter in series with the lamp, or ditching the lamp entirely and going with a 150 W heating element, which won't exhibit the huge changes in resistance between cold and hot.Sometimes it is the seemingly simple components (like relays and light bulbs) that will bite you in the butt if you aren't careful...

author: madires
author_rank: Super Contributor
timestamp: April 30, 2014, 01:33:24 pm
content: Quote from: chipwitch on April 30, 2014, 12:38:52 pmIf the relay contacts are welded together (by overload, arcs or wearout) they stay that way even if the power is turned off. Since everything's fine again after a power cycle the problem is caused by something else. I'd suggest to add some extra sanity checks to the code to prevent overheating and to use the ATmega's watchdog in case the MCU goes beserk. Another issue might be that the load switching causes some trouble for the MCU (better decoupling, snubber etc.).

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 01:55:19 pm
content: Quote from: N2IXK on April 30, 2014, 01:30:27 pmDouble checked the relay... not tungsten rated. The plan was to get away from incandescent bulbs ASAP. Had been thinking about alternatives. I have a cartridge heater, 250VAC 500W that I was going to use (at 120VAC), but I hadn't gotten to it yet. Also, I was thinking about repurposing a hair dryer heating element, stepped down via a transformer. Would I be right in assuming that kind of setup the concern would be in breaking contact? Hair dryer elements are nichrome, right? Do they react in the same way as tungsten?

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 02:02:26 pm
content: Quote from: madires on April 30, 2014, 01:33:24 pmThey aren't welded... not fully in any case. I should have disconnected the Arduino to see if the relay opened. If it had, it would mean there was no welding at all. But, I didn't do that. I just unplugged everything without assessing first. I do know that when I plugged everything back in, the temperature had dropped to well below turn-on so obviously the light was on. When it reached turn-off, it turned off as expected. Snubber may indeed be the problem. Considering that the incubator ran 24/7 for 3 weeks before failure, I figured we were safe. Goes to show ya.

author: calexanian
author_rank: Super Contributor
timestamp: April 30, 2014, 02:13:42 pm
content: You want to make sure that relay coil has a free wheeling diode across it. Very important. Also a higher contact rating would be nice with an appropriately sized MOV across the contact. In practice relays are very reliable. We use 45K plus a year and have failures countable on two hands of the actual relays themselves. We even make products in the poultry industry.

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 03:08:32 pm
content: Quote from: calexanian on April 30, 2014, 02:13:42 pmIt's been a few months since I designed the circuit so I had to take another look. Snubbers are something I only learned about recently, so I couldn't remember if I included that in the relay circuit. I did. There's a flyback diode on the coil. At least I got that much right.No doubt I cut corners to hasten use. I had the eggs and times was of the essence. The eggs would have been destroyed otherwise. MOVs and other protections would have been much appreciated. I had all the "jelly beans" except fuses and MOVs. I now have an assortment of buss fuses. I probably would have considered a fuse to the relay contacts unnecessary. I won't make THAT mistake again. (yeah, I probably will). But why the MOV across the contacts? I would think if you're going to use voltage protection, you'd use that closer to the power supply and protect where power comes in from the mains?

author: richard.cs
author_rank: Super Contributor
timestamp: April 30, 2014, 03:23:43 pm
content: Quote from: calexanian on April 30, 2014, 02:13:42 pmBe careful with flyback diodes across the coil. The simple single-diode doesn't allow a particularly quick decay of the stored energy in the coil and can cause the contacts to open too slowly and arc badly. A diode with a series resistor or an antiseries zener is better, the back-emf is still limited but to a rather higher value and the contacts will open more quickly.Being fixed with a power cycle suggests it's not the relay, but in general a correctly selected relay will be very reliable. I'd second the two in series suggestion with the proviso that it doubles the chance of the lamp not coming on (whilst squaring the probability that it will stick, a much greater reduction). Feedback of the actual heater status would be good.

author: Conrad Hoffman
author_rank: Super Contributor
timestamp: April 30, 2014, 04:50:22 pm
content: If the application isn't crazy cost sensitive, I'd be using a solid state relay and the above mentioned MOV to protect it. I'd also want an entirely independent safety system like a thermoswitch to prevent ever going too high. If you could establish a baseline power need, to just keep the system near the right temperature, without going high under high ambient, you might install a fixed heater to insure the system never drops too low. The active system can then be designed for less power, plus if it fails you have far more time to take action.

author: chipwitch
author_rank: Frequent Contributor
timestamp: April 30, 2014, 05:28:58 pm
content: Quote from: Conrad Hoffman on April 30, 2014, 04:50:22 pmThat's not a bad suggestion but cutting close to the bone is more work than just turning the source on and off as needed. Plus, many factors can influence the temperature. I would be concerned about such a method's effect on hysteresis. I'm not fond of heater cartridges for that reason as well. A light bulb or heating element will cool down very quickly without overshooting the target temperature due to little mass infusing more energy into the system after turn-off. Not to say that it wouldn't be better doing it the way you suggest, but it would require more effort in ensuring proper balance.

author: pickle9000
author_rank: Super Contributor
timestamp: April 30, 2014, 06:24:58 pm
content: If you are thinking in terms of safety (medical or even household appliances for example)- detect the actual light coming on and off, a photocell can do that with little effort.- a thermal cutout to disconnect power to the bulb if the temp goes too high (often used in portable heaters if you need to steal one). These come in many different temp ranges and only cost a couple bucks.- detect the temp change in software, if you don't see the temp going up or down when it's supposed to then sound an alarm. Perhaps have a second relay to completely disconnect the power.- Set up an auto reset function. A timer that will hit the reset on the micro if not reset every minute or so. If the micro stops hitting the timer reset the micro will be forced to be reboot. Some micro's have this as a built in feature.

author: kaindub
author_rank: Regular Contributor
timestamp: May 01, 2014, 01:03:52 am
content: I think you actually have not found the root cause of your problem.In my experience (many years of control systems using relay logic) relays always failed to close (not failed to open) if sized correctly.Be aware that as someone has pointed out , relays have a number of ratings. The simplest is the thermal rating of the relay. This is basically the amps that the relay can carry. It assumes no switching.The second is called an AC1 rating. This is the rating of the relay in amps and volts (yes amps and volts) that a relay can reliably carry. It assumes a certain switching frequency and characterises the lifetime of the relay . Lifetime is usually about 1000000 switching cycles.There are also AC2 and AC3 ratings. AC2 rating is for switching lamps and AC3 is for switching motors.Its surprising how low a rating a relay gets compared to its thermal rating for AC1, AC2 and AC3 ratings.I imagine that your controller has some sort of control that causes the lamp to go on and off . Make sure that the relay is not chattering (ie switching too frequently). Remember that the thermal time constant for your system is in the order of minutes which is relatively slow. So trying to regulate to a narrow temperature range is self defeating.I think the relay is undersized and switching too frequently and just wearing out. The solid state relay may be an answer to your problem, but again it needs to be sized correctly. Solid state relays go short circuit when they fail, so you'll end up with hard boiled eggs.Robert

author: sleemanj
author_rank: Super Contributor
timestamp: May 01, 2014, 02:43:04 am
content: A few years back I slapped together an incubator, a bunch of muscovy ducks emerged at the end of it (100% hatch rate), somewhat amazingly since they had a VERY rough ride with many extreme temperature and humidity excursions at both ends of the scale, there was no automation, just a light bulb, a dimmer, and a spray bottle of water. Built into a polystyrene bin with glass lid and polystyrene cover, I did stick a CPU fan in (running very slowly) to move the air around a bit and even out the temperature.The bulb I used in the end was a simple 25W candle bulb from memory, it produced the best amount of control across the dimmer range. It was a small box, I guess about 30cm long, by 20cm wide, from memory I hatched 8 at a time in it and maintaining the temperature was easy with the bulb just dimly lit. It was important however, even given the polystyrene construction, to have the incubator in an area which was fairly temperature stable and where there would not be any random breezes.Anyway, if (when) I do it again, from the lessons I learned, I would also make it automated, both for temperature and humidity.I'd probably drive whatever heating device with very slow frequency (as in cycle per minute speeds) PWM DC simply, using a mosfet, the temperature I wouldn't control too tightly as long as it's in a band of a few degrees it will work. As long as the incubator is well insulated (I'd use a polystyrene chilly bin again) the actual power you need to drive is small, it will take a little while longer to get to temperature, but that should be OK, in the real world eggs are subject to plenty of temperature variations just from the hen moving them, going off to feed...The humidity I would use a small water pump with a spray nozzle to introduce humidity, and a very small side mounted fan to extract it, again, not controlled tightly, just check every few minutes and spray a bit, or vent a bit depending on the value.I'd have a battery powered and completely separate "watch dog" which would just beep if temperature went out of spec, battery powered because I want it to alarm if the power goes out, or whatever. Doesn't have to check often, once every few minutes is fine, then it can go back to sleep unless something bad is happening. So battery will have no problems lasting the incubation period.Once hatched, they move to a brooding box which would just remain low-tech, a strong bulb, with plenty of room for them to move closer or further from it as they wish, copious food, water... and cleaning.PS: I highly recommend adding some muscovy ducks to your menagerie, we have various chickens and bantams too, but the muscovy have real character (and prolific large eggs). Unlike other ducks they don't make noise (they don't quack), although they will dig up your lawn if it's wet and soft, hide nests in impossible to find places, sneak inside to eat the left over cat food when nobody is looking, and go flying around the neighbourhood if you don't clip the wings before they realise their new batch of feathers have grown in.

author: Richard Crowley
author_rank: Super Contributor
timestamp: May 01, 2014, 03:07:10 am
content: Any kind of resistance heating device is near enough to "tungsten" that you can consider it equivalent.A dirty environment like that is no place for an ordinary relay, even one of adequate rating.SSRs are so cheap, there is no practical reason NOT to use them.

author: richard.cs
author_rank: Super Contributor
timestamp: May 01, 2014, 04:54:09 pm
content: Quote from: Richard Crowley on May 01, 2014, 03:07:10 amThat's not really true. Any radiant heaters, yes, but forced-air heaters and (especially) water heaters operate with low element temperatures and so the change in resistance is much less. Tungsten is worse than most heaters, but by a widely varying amount.

author: Richard Crowley
author_rank: Super Contributor
timestamp: May 01, 2014, 06:53:45 pm
content: Quote from: richard.cs on May 01, 2014, 04:54:09 pmSo, within THIS context, yes radiant heating schemes (in non-specific, general, hobby-level terms) can be considered equivalent for casual "system engineering" as chipwich has demonstrated here. Of course, there are variations, but safer to assume high inrush currents unless/until proved otherwise. This specific example is such a great place to use an SSR, I don't know why people even use those old clunky mechanical things anymore?