What is wrong with my home automation project? - Page 1
Forum section: electronics / projects

author: EtaPhi
author_rank: Contributor
timestamp: April 25, 2021, 03:54:22 pm
content: Preliminary remark:I developed the uAnts4Home project in my spare time.I built a working prototype in 2008.The uAnts4Home project (pcb, schematics, firmware and management software) has been available from SourceForge since 2009, so all information is public domain.I don't want to sell anything because everybody can build his/her copy.I only want to understand what is wrong with its structure which I'll briefly explain.uAnts4Home is a home automation system which is made of many small, cheap and interconnected devices called uAnts (micro ants).Each uAnt has got inputs, a state and outputs.Inputs, states and outputs may be digital or analogue according to uAnt configuration.uAnts alone can automate simple things, like switching on/off a lamp.A uAntColony (micro ant-colony) is established when they are interconnected to a uAntHill (micro anthill) by a communication media.The first prototype used a RS-485 bus, but a radio or plastic optic fibers links can be used (then there was no ESP8266 module nor any cheap opto component like nowadays).The uAntHill is alike an orchestra director: it does not play, but gives timing to operations.Its role will become clearer (I hope) when I'll explain the uAnts Protocol.A uAntColony, being a set of uAnts, has got inputs, states and outputs too.Inputs and outputs are local, i.e. are managed by the uAnt they are physically attached to.A set of equations models input - output relationship:X[k+1] = F(X,U)Y[k+1] = G(X)where:* U is the input value at time k,* X is the state at time k,* Y is the output at time k,* F is the state transition function,* G is the output function.Details of F and G are not important for this thread, as anyone who passed a course about Control Theory may explain you them.I want instead explain the previous model effect on uAnts.F() allows each uAnt to update its state when it knows its inputs (true, because they are local) and the uAnt Colony state (it may be a subset of states, but things don't change).G() allows each uAnt to update its outputs when it knows uAnt Colony state.If the above is true, every computation may be done locally if each uAnt broadcasts its state so that any other uAnt can read it.Every programmable logic controller (PLC) uses a discrete time model too, which is coded as a Ladder Program or a Function Block Diagram or Sequential Function Chart or Structured Text or Instruction List as every field engineer can tell.There is only a difference: in a PLC every computation is performed by the central processing unit (CPU) whereas in a uAntColony processing is local.As a bonus, there is no need to acquire remote inputs (one control network access) and to actuate remote outputs after processing (another control network access), but there is only a network broadcast to propagate states.The uAnt Protocol deals with uAntColony state propagation, which happens every control cycle. The uAntHill starts state propagation by sending a special signal.In 2009 prototype, this signal was a break (13 space symbols) which allows uAnts local clock frequency adjustment too so that no quartz resonator is needed.Every U(S)ART I know can send and detect breaks, so no special code must be written.After uAntHill state propagation signal, every uAnt broadcasts its state.The next start signal (i.e. break in RS-485 bus) will end a state propagation cycle and start a new one.This signal will also make uAnts update their states and outputs.The uAnts Protocol defines how states are coded and what uAnts must do when they receive uAntHill start of state propagation signal.A short document explains its details for RS-485 bus which can be summarised as follows: the uAntHill sends a byte (i.e. the uAnt address to poll) and the corresponding uAnt sends the required byte of its state.The uAntHill uses the reserved address 0 to tell which byte it wants to read, i.e. will be broadcasted, since all uAnts may listen the selected uAnt response.For instance, when uAntHill sends bytes 0x00 0x12, uAnts know that they must reply with the byte at address 0x12.There are also byte sequences for uAnts remote setup, which are alike to uAnt state write, but they aren't meaningful in state propagation.Wireless links may use UDP packets (maybe encrypted for increased security) to propagate states, whilst uAnts may attach a payload with their state to uAntHill start frame in plastic optic fiber links which are usually daisy-chained.I apologize for this long description of maybe obvious things.I however don't realize why simple and well known concepts aren't used in home automation systems.A uAntColony shows that a home automation system without commands is feasible, but nobody cares.Drivers, i.e. commands translators matters so that a home automation is said better than another if it has more drivers.Commands add complexity, because who sends a command must know what commands a device accepts.Commands must be secure, because who executes one must be certain that what it received was sent by an authorized device.On the contrary uAnts Protocol is a matter of state byte reads or writes whose meaning is known.No special driver, no expensive hardware is needed.A cheap 8 bit micro as STM8S003 or a cheap ESP8266 module has enough resources to build a uAnt.A ESP8266 module can also bridge a RS485 bus with a wireless link, since it has got a spare UART to send and receive bytes.Plastic Optic Fiber links are simple too, since their use needs an input capture plus an output compare channel (STM8S003 has 2 timers and 4 + 3 channels).I'm sure that my new prototype built with STM8S003, ESP8266 (or equivalent modules) and plastic optic fibers will work, but I'm not sure that something will change in home automation as long as command based structure will last.Please, tell me what is wrong with uAnts, because after I built and tested first prototype, no other automation system seems interesting to me.

author: Renate
author_rank: Super Contributor
timestamp: April 25, 2021, 07:33:34 pm
content: That's a good question.It's hard selling (even if you're not really selling) somebody on a system concept.You have to make it clear how it works. You have to make it clear how it's better than something else.Your description above sounds abstract and I'm not sure what's going on.Using a new protocol requires some justification.I have an automation system. Monitoring and logging plays a bigger part than commands.I guess that there is very little "automation" as in "if X, then Y".Configuration is "compile time" (which would be hard to sell to others) but works fine for me.I use Modbus at 115.2kThis is integrated with an off-the-shelf solar controller which already uses Modbus at 115.2kA lot of people get sucked into the Modbus vortex because of that.Your statements on xtal-less operation and sending breaks makes me uncomfortable.Maybe some of the problem is just documentation?I didn't see any link. (Although it was easy enough to find.)

author: JohnnyMalaria
author_rank: Super Contributor
timestamp: April 25, 2021, 07:37:21 pm
content: Quote from: EtaPhi on April 25, 2021, 03:54:22 pmBecause the giants in this space rely on proprietary (usually patented) technology to create a closed ecosystem.

author: EtaPhi
author_rank: Contributor
timestamp: April 26, 2021, 03:44:05 am
content: QuoteYou are right, but patents expire after 10 years.That's why I can affirm that uAnts are patent-free: SourceForge publication, a book I published about uAnts and a public seminar I held in 2009 all create evidence of prior art against which no patent claim can be enforced.Closed formats and ecosystem are better for big corporations, because never expire and there are consumers that buy a component that cost more and may not be replaced when it will break since there is a better one (but it's also expensive).The iPhone is such a product.HomeKit is Apple effort in home automation arena.If you wanted to build something which is compatible with Apple hardware, you had to buy a chip from Apple, i.e. pay royalties to them.Now it seems that this is no more needed, since even Apple can't create a closed ecosystem.One of my errors was a communication one.Tech people are more geared to technical details than telling stories to buyers.In 2009 there weren't available the tools and technologies I'm now planning to offer to whoever adopts uAnts.For instance, no 3D printer was available.A bare working board is enough for techies, but people want a finished product.uAnts may be a reason to buy a 3D printer, a tech toy which many wants but can't justify because it's useless if you can't create g-codes.A 3D printer is useful for uAnts, because you can build a spare part when something breaks.It's a longevity assurance.One of closed ecosystem issues is replacement parts availability.Think about a ten or twenty years ago product.Some were produced by a now acquired or bankrupted firm, some products are now discontinued, some have expensive spare parts.Open source systems are better, because I can now build a replacement part with existing components.For instance, uAnts used ST7FLite29 and ST7FLite39 micros which now are expensive, but I can use a 40 cents parts as STM8S003 to build a new uAnt which is 100% compatible because of public communication protocol specification.I plan to go further: a tech person with some tools and manual skills will be even able to build pcb with a cheap laser engraver.These tools aren't fast enough for mass production, but are good for tech people who want to service their home automation system.So, yes I was not successful because of communication.Now I have a story to tell.Perhaps I sold you no uAnt (yet), but things may change...

author: james_s
author_rank: Super Contributor
timestamp: April 26, 2021, 03:59:29 am
content: The secret sauce is the software/interface that glues it all together.There are open source home automation systems though, I've been using Home Assistant but OpenHAB is another popular option. The community forum for Home Assistant is completely useless but the platform works reasonably well and there are lots of youtube videos made by other people on how to set things up. It works with all sorts of different hardware from various automation systems without requiring the proprietary bits.

author: EtaPhi
author_rank: Contributor
timestamp: April 26, 2021, 04:36:19 am
content: QuoteOne of big issues of command based home automation systems is "commands pollution".Commands are easy to sell, as the current voice control tide shows.It's easy to advertise a smart socket which switches on or off speaking to your smartphone, but this an enhanced remote control, not an automation system.When complexity grows, so happens to command limitations.For instance, if one wants to switch on home appliances, such as air conditioners, when sun shines and his/her PV system produces energy, what he/she have to say to his/her smartphone?I wrote "commands pollution" because system logging requires devices states which can be read by a command.These state read commands are the majority in a working system.If communication media is not fast enough (I think of KNX protocol), system performance degrades.Modbus is great for logging and supervision, but uAnts Protocol is even better because, as I wrote in a document that you can find in uAnts4Home site, fewer bytes have to be transmitted to have devices states.As regards break signaling of uAnts RS-485 protocol and xtal less devices, I was concise to keep introduction small.It's easy to explain these concepts.Serial communication needs a clock signal to acquire bit values.A 115.2k speed means that a bit must be read every 8.68 microseconds from the start bit.A RS-485 bus is in high state when it's idle.A start bit is represented by a low state for at least 8.68 micro seconds.When an UART detects a falling edge, i.e. a start bit, starts bit acquisition according to a well known specification which mandates a +/- 2% frequency difference (if I remember well) between transmitter and receiver clocks.External crystals are an added cost and are a noise source and failure point which modern micro avoid with their internal RC oscillators.There is an issue: some are not stable enough for serial communication (e.g. USB, or RS-485).Some manifacturers have a solution: trimmable RC oscillators, i.e. firmware can adjust RC frequency by changing an internal control register.The break signal is also useful to trim a RC oscillator.You know that a break signal lasts 112.8 micro seconds @ 115.2k bus speed, so a microcontroller can measure its duration and adjust accordingly RC trim control register.If it seems to last more, RC oscillator frequency must be decreased; if it lasts less, RC frequency must be increased.Only the uAnthill needs a crystal because it's the only device which can send a break signal which is longer than any transmitted bit sequence (1 start bit, eight data bit, one parity bit and one or two stop bits).This allows uAnts cost savings, which is always good for consumers.

author: EtaPhi
author_rank: Contributor
timestamp: April 26, 2021, 05:51:56 am
content: QuoteSorry, Home Assistant, Open HAB etc are indigestible sauces for uAnts...They are a patch to command-based home automation systems.I mean, product X can't use product Y because their protocol differs.One therefore uses such systems to "translate" commands from X to Y and viceversa.uAnts use states, not commands, so Home Assistant or Open HAB would have states from uAnthill and could not send commands.Of course, it's possible to encode commands as state bits (e.g. set bit 4 of address 2 in uAnt 14 to force the state which switches on a lamp), but it will be a mess.Command based systems are good for their manufacturer which force you to use their protocol.Tons of device drivers, translators, additional hardware are needed to integrate devices so a home owner prefers to stick with a vendor to avoid this mess.State based systems are vendor neutral, since force manufacturers to use a common protocol.Which system do you prefer?

author: Renate
author_rank: Super Contributor
timestamp: April 26, 2021, 11:12:02 am
content: Quote from: EtaPhi on April 26, 2021, 04:36:19 amWell, I just don't get it.If you want reliable serial communication, throw a crystal on it and you are done.Yeah, I do have this junky box that bit bangs serial out of an SPI with an internal RC oscillator.But that's an 8 pin DIP ATTiny85 that I threw together.The point is, if we don't see eye-to-eye at the start on little details like this, why would I invest the time to even look into your system?I could literally spend the rest of my life just doing a comparison between home automation systems.Instead I just built one.Do I think mine has universal appeal? Nope.If I have a philosophical axe to grind it would be:Wired has some advantages over wireless.Modbus is a reasonable choice unless you want to go to CAN.Access should be by web page, API (app) and ssh.

author: EtaPhi
author_rank: Contributor
timestamp: April 26, 2021, 02:10:55 pm
content: QuoteDid my words irritate you?I hope not. If it was so, my apologies.I respect your opinions and understand that experience matters.As regards a silly crystal which may be placed or not in a pcb, do you prefer to throw the baby with the bathwater?I don't want to design a circuit without crystal, I simply said that it's possible to run a micro from its internal RC even if its frequency must be precise enough to talk over a serial line.I also believe that internal oscillators have got some advantages, so it's up to the builder to choose between installing a crystal or not.Moreover, STM8S has got a clock security circuit which may be missing in ATTiny85.When the external crystal fails, it automatically switches on internal RC oscillator to recover from this failure.This matters for me, as I have a SOMFY motorized shutter which is stuck closed because of a failed crystal of its wireless remote.Details are important, but design philosopy counts more, IMHO.You prefer wired links to wireless ones.Modbus or better CAN over other phisical layer protocols.Web access with an API or app or ssh terminal to interact with your home automation system.I mostly agree with you, but my top preference is state based systems instead of command based ones.Since I cannot find other state based systems, I wonder if my opinion is right or wrong.I asked this forum what may be wrong in my home automation philosopy.I'm not finding followers here, but I'll be glad if I find one.I'll be also glad to respond to your questions, if you want more informations.If you don't like my home automation philosopy, I have no problem because I think too that my home automation system has got no universal appeal.

author: JohnnyMalaria
author_rank: Super Contributor
timestamp: April 26, 2021, 02:39:16 pm
content: Quote from: EtaPhi on April 26, 2021, 03:44:05 amMore like 20 years. Plus new patents come along every few years with new versions of the technology. Few vendors are interested in an open source protocol. They intentionally want you locked into their ecosystem.But I can't comment on the merits of any particular home automation system. I'm too old and my house is too small to bother

author: Renate
author_rank: Super Contributor
timestamp: April 26, 2021, 02:56:06 pm
content: Quote from: EtaPhi on April 26, 2021, 02:10:55 pmNo, we're just talking about stuff here.I (somewhat) understand what you are saying about state, but everybody uses state.But I don't understand why you're being so formal and abstract.My HVAC control has mode, low-set, high-set and current state.I could express it in terms of F(t).It's actually more complicated with fan run-up and system cool-down.But in the end it's a dozen lines in a subroutine.I don't even model the stored heat in the furnace.When you shut the burner off the room temperature increases above the "high-set".If I modelled exactly I could have the peak temperature hit a prescribed temperature.But in that case I'd just have to set the "high-set" a bit higher so that I get the correct time cycling (~3x / hour).In any case, I'd rather just work on my status display.

author: phil from seattle
author_rank: Super Contributor
timestamp: April 26, 2021, 03:14:09 pm
content: You ask "What is wrong".Simply put - the way you talk about it.Even technical people need to understand the benefit of using a piece of technology. You may see it very clearly in your mind but if you can not cause others to see it, you will get no where. In reading your first page of writing, I saw one sentence "switching on/off a lamp" that went to actual benefit. You talk about protocol and state transitions but do not show how that makes your system better than any others.Cheaper, smaller, faster, more reliable, easier to use, easier to set up,... If you can not show any of those within a few paragraphs, you will never be successful.

author: Bassman59
author_rank: Super Contributor
timestamp: April 26, 2021, 03:45:20 pm
content: Quote from: EtaPhi on April 26, 2021, 02:10:55 pmThe neat thing about asynchronous serial ports (RS-232, RS-422/485) is that they re-synchronize with every byte. The oscillator frequency has to be pretty far off for it to fail. The 2%-spec oscillators in the Silicon Labs micros is "good enough" for 115,200 bps RS-232.Of course things like USB and Ethernet require much more accurate oscillators.

author: Bassman59
author_rank: Super Contributor
timestamp: April 26, 2021, 03:49:14 pm
content: Quote from: EtaPhi on April 26, 2021, 05:51:56 amExplain to me how you set or get a state ... I assume that you must send a command to the device?How does a main controller learn the state of a device? That is, I have an app that I use to control my gizmos. When I start that app, it needs to learn the state of everything on the network. Normally, it would send out status-request commands to known devices.It also has to discover new devices, so again it must send some kind of broadcast command.

author: Bassman59
author_rank: Super Contributor
timestamp: April 26, 2021, 03:50:44 pm
content: Quote from: james_s on April 26, 2021, 03:59:29 amOh, god, Home Assistant works, but its user interface is daft and yeah, the support forum is useless. Entities vs devices, really, what the fuck is the difference?

author: EtaPhi
author_rank: Contributor
timestamp: April 26, 2021, 07:54:29 pm
content: QuoteStates are stored in a byte array.Let's say that each uAnt state is stored in four bytes.uAntHill, the main controller as you call it, cyclically scans uAnts state by sending a byte sequence to read each state byte.It sends 0x00 0x00 sequence to start byte 0 scan.Then it sends 0x01 and waits for byte 0 value from uAnt at address 0x01.If there is no answer for 2 byte periods, assumes that uAnt at address 0x01 is busy and goes on with next address, i.e. sends 0x02 and waits for byte 0 from uAnt at address 0x02.This byte 0 scan cycle ends when there are no other uAnt to probe (more about this later).After byte 0 scan is completed, it sends 0x00 0x01 sequence to start byte 1 scan; it sends 0x01 and waits for byte 1 from uAnt at address 0x01, and so on for the remaining uAnts and state bytes.uAnt discover and address assignment need no special command.When uAnthill has to discover new devices, it emits one more address during its scans.Let's say that there are ten devices. A valid address is therefore a number between 1 and 10.If uAntHill is in "discover mode", each state byte scan will end at address 11 instead of 10.Since every request to device 11 is unanswered, every unpaired uAnt knows that address 11 is free.Note: an unpaired uAnt is a device without address which can listen but not answer.When the installer pushes a button of an unpaired uAnt, it takes the free address and starts responding to uAntHill requests.These new responses tell uAntHill that a new uAnt joined uAntColony, so that it updates uAnts count and sets the newcomer cryptographic key which is used to unlock its configuration.The installer action is needed to solve assignment conflicts that happen when there are two or more unpaired uAnts.uAntHill "discover mode" may be disabled for security reasons, so that no other foreign device can acquire an address and talk.As you can see, no special command is needed for pairing.uAntHill (and every uAnt) may store all states which are immediately available to external devices if uAntHill has enough RAM, or can be obtained in a scan cycle.This is a feature of uAnts protocol: normal operation and special functions don't interfere and performances are granted by design.uAnts Protocol has write state sequences too, which are those sequences of type 0x00 0xNN where 0xNN is a number between 0x80 and 0xFF.If uAntHill wants to write 0xAB at state byte 0x00 of uAnt at address 0x12, it will output the following sequence: 0x00 0x80 0x12 0xAB.As you can see uAntHill may access up to 128 bytes at the same time, i.e. four pages of 32 bytes.A bank switch register allows to increase total addressable memory of uAnts to 255 x 32 bytes since page 0 is always mapped at the lowest indexes.A uAnt doesn't immediately execute write sequences, since there is a security lock which prevents unallowed changes.A challenge-response trial must be passed, i.e. uAntHill must read a challenge from a crypto register which is allocated in page 0. Then it has to compute a response and write it to crypto register. If uAnt and uAntHill responses match, write sequences take effect.

author: james_s
author_rank: Super Contributor
timestamp: April 26, 2021, 08:00:55 pm
content: Quote from: Bassman59 on April 26, 2021, 03:50:44 pmI gave up worrying about that, the developers are completely uninterested in any sort of constructive criticism, their attitude seems to boil down to one of two basic responses, either "you're an idiot, read the docs" (which are the worst and most useless of any open source project I can think of) or "it's free, if you don't like it, write your own". The forums are filled with people asking questions and lots of others chiming in saying essentially "I have the same problem" and very few answers. Half the time when I tried asking for help on how to do something they seemed more interested in arguing with me over why I want to do whatever it was I was trying to do.Well, whatever, ultimately it does work really well now that I got it all set up. I took the same approach I took with KiCAD in realizing that all available EDAs are buggy crap with weird interfaces and steep learning curves and I just picked one and forged ahead until I learned to work within its warts. The actual underlying framework is fine though, mqtt is the closest thing to an open standard for home automation that we have. I see no need to develop some entirely new replacement for that part.

author: ajb
author_rank: Super Contributor
timestamp: April 26, 2021, 08:43:22 pm
content: I've been pondering some home automation stuff for a while, and have been kind of lightly surveying options including home assistant, but haven't taken the plunge yet.The thing that initially strikes me as a potential drawback with the system described here is that if the individual devices are responsible for holding the logic that drives their actions in response to input, then that means that the total configuration of a home automation system gets spread out across all of the constituent devices.Every device has to be able to understand what a particular system state means, and what it should do when in that state. The more devices you have in a system, the more fragmented the configuration becomes (or the more it must be duplicated if many of those devices need to do the same things). The only sort of application where I think this has any advantage is one where most devices have direct access to most of the information they need locally--for example if you put a thermostat with a built-in temperature sensor on this network, the thermostat can do most of its job in the absence of information from the other devices, which maybe improves its reliability or the ability of the overall system to degrade gracefully. However I'm not sure that most home automation projects are like that. One of the real problem-solver selling points of home automation is that I can stick a smart outlet here, a smart wall switch over there, maybe an occupancy sensor in that spot, and then once they're linked to my automation hub I can add a new behavior in the same place I've created all of my other behaviors and get new functionality. I don't need to configure the smart outlet with the necessary logic to do what I want, and if I want that new smart switch to also control another existing smart outlet I don't need to go over to that other device and also change its configuration--it's all done in one spot. It's also harder to replace components, because I have to reconfigure each one I replace to give it the correct logic/whatever, and if I replace a device that other devices use as an input I potentially have to reconfigure those devices as well. Especially with a non-isolated wired network, where an electrical fault or (or close enough lightning strike) could wipe out the entire network in one go.A more conventional centralized system in general is going to be easier for most people (regardless of skill/technical level) to manage because all of the configuration can be in one place. It's also potentially easier to backup and restore. It also doesn't preclude having local logic where that makes sense. Like the thermostat example, where it can have as complex an algorithm as you want, but still allow an external control system to change modes, setpoints, etc.The other big thing that makes this sound less appealing, which may just be the way you've chosen to present it here, is that there's no mention of abstraction. Regardless of how technical your audience is, talking about system state as a string of hex values really does not make it appealing. I don't want to have to remember that device 0x37 is the attic fan controller and that state 0x03 means that it's currently running the fan, I want a nice user-friendly display that says "Attic fan: ON", and I don't want to have to jump through a bunch of hoops to get that. I also don't want to have to remap all of my devices on the bus because I needed to rearrange things and now all of the device addresses don't match the configuration that's spread across my network. Maybe your system has that, but it's not apparent from the description here. Really, the processing load for most home automation tasks is so low that even the cheapest MCUs can spare enough cycles to generate and parse messages that provide useful context--if not directly to the user, then at least providing a useful description of themselves and their capabilities to the controller which then presents them to the user--so there's really no good reason to NOT provide that ability when it would make setup and use of a system much easier.

author: EtaPhi
author_rank: Contributor
timestamp: April 27, 2021, 04:34:25 am
content: QuoteYes, uAntColony configuration is stored where it is needed, but this is a "low level" view.uAntHill provides a "middle level" view, and a "high level" view is given by an application that runs on a PC.If uAntColuny were a CPU, low level view would be its ALU and registers description; middle level view would be a debugger or a compiler and high level view would be an IDE for program development in your favorite programming language.uAntHill is a single control point. A PC program or an application can read and write uAnts working logic so that the only action that is locally needed is a button push to set each uAnt address. There are no dip-switches, there is only a "write protect" device (it's a potentiometer in 2009 prototype) which prevents any modification, even uAntHill ones for maximum security.QuoteAn Integrated Development Environment (IDE) allows an user to give a name to inputs, states and output and "configure" (or program) uAnts.In PLC (Programmable Logic Controller) jargon this is the I/O mapping.In MCU programming this is register mapping, i.e. TIM1ARR_1 (timer 1 auto reload register 1) is mapped at address 0xNNNNNN (see your favorite MCU datasheet for a real example).When you compile or assemble a MCU program, these symbols are converted to their numeric values and stored in executable binary which will be "programmed" in MCU non volatile memory.My previous description was low level, where there are only bytes, offsets and addresses.At an higher level there are named inputs (e.g. Attic Thermostat Winter/Summer switch), states (e.g. Attic Temperature, Attic Fan Winter/Summer switch position [Winter or Summer], and outputs (e.g. Attic Fan Coil) which are defined in I/O mapping.A programmer defines control logic in a high level language (it may be ladder program, o whichever language IDE provides) and a "compiler" translates this program source in an executable that uAntHill downloads to uAnts.I hope that my explanation of uAntColony basics is now clear.It's my fault if I wasn't clear from the start, but to my excuse I find hard to explain a system which is like a distributed and parallel PLC by using words that every housewife can understand.

author: 2N3055
author_rank: Super Contributor
timestamp: April 27, 2021, 07:02:01 am
content: Quote from: EtaPhi on April 27, 2021, 04:34:25 amThat is answer to your question..

author: Bassman59
author_rank: Super Contributor
timestamp: April 27, 2021, 03:31:38 pm
content: Quote from: 2N3055 on April 27, 2021, 07:02:01 amAnd there it is ... for the system to gain widespread acceptance, it has to be understood by people with no technical training.

author: Bassman59
author_rank: Super Contributor
timestamp: April 27, 2021, 03:59:45 pm
content: Quote from: EtaPhi on April 26, 2021, 07:54:29 pmIn other words, the main controller sends commands to each address and waits for a response.QuoteYou say "sequence of bytes," I say, "command," let's call the whole thing off!*QuoteOr, you send it a command, and wait for a valid response ...I'm not busting chops. I just see this as an overly complicated network scheme. I agree with the idea that each device maintain state, but isn't that what these things do now?For example, say you have a dozen controllable switches and lights in your home. Most of the time, no controller is connected. And the Ikea switches and lights can be controlled by little gizmos that talk directly to the things, no computer or network involved. This means the state of each device can change independent of any kind of main controller.We all have apps on the phones and programs on the computer to control our gizmos. Those apps/programs never maintain a constant connection to the devices, so when you start them up, they scan the devices they know about to determine their state. (They do that by ... sending commands and getting responses.) They can't assume the devices are in the same state now that they were in yesterday. Adding new devices to your set-up requires special interrogation commands, which you don't want to run all the time.What's great about Home Assistant running on a Raspberry Pi is that (despite its awful user interface and asshole developers) is that it keeps track of the state of all of your things, so there's One App (or web browser) to go to that gives you access to all of your things, regardless of vendor. Open Home Assistant and on your main display you have all of the things and their current state.All that said:I don't see what wrong with standardized commands like "Light On" or "Brightness Level 50%" or something. Hell, even sending MIDI messages to the devices would work, as long as the messages are standardized and documented. The problem is that all of the device manufacturers cook up their own commands and then wrap (quite reasonably) security around the whole thing. Then some of them have their cloud bullshit, which is the REAL disaster.

author: james_s
author_rank: Super Contributor
timestamp: April 27, 2021, 05:17:53 pm
content: The "cloud bullshit" is the primary thing that pushed me away from all of the commercial offerings. Next is the proprietary nature of each system, there is no standard to ensure that devices can work together with other devices, it's the "Wild West" era of the personal computer all over again. There needs to be a standard, such as mqtt, and I agree that there should be a large selection of standardized commands, along with the ability for manufactures to define custom commands. Home Assistant (and others) have the right general idea in that they allow all this different stuff to work together. It would be nice if somebody sold a product that was a hub running a fork of that all preconfigured for a turnkey solution similar to the commercial products but with the flexibility to be customizable.

author: EtaPhi
author_rank: Contributor
timestamp: April 27, 2021, 09:43:11 pm
content: QuoteIt doesn't matter if you call a sequence of bytes a command, because what is really important is that you understand what's going on.These bytes are low level information that is broadcasted.They are alike CPU machine code, for instance the STM8 assembler instruction:Code:LD A,#$12may be seen as a command or the byte sequenceCode:$A6 $12that assembler emits when processes it.QuoteThe challenge/response security protocol is an example when commands and byte sequences differ slightly.The uAntHill does not wait after every byte of its response is written to uAnt crypto register, but as a side effect, if uAntHill and uAnt results match, all write operations take effect.A similar situation happens to condition code register when STM8 CPU executes the above assembler instruction, i.e. Z and N flags are set to zero.Is LD A,#$12 a command to clear these flags in CC register?I don't think so.These are however details, while the main, outstanding, feature is neglected: if "commands" are translated as write or read sequences to some addresses, new commands aren't nedeed.For instance, the uAnts challenge/reply security protocol adds no new command as they are defined by a sequence of "atomic commands" i.e. read byte / write byte.I think that uAnts Protocol is a solution to integration headaches of products from different manufacturers, but it's my humble opinion.MQTT is another solution which is somewhat alike.It's indeed easy to transform uAntHill interface with external world into a MQTT server, because they both deal with states.

author: Vovk_Z
author_rank: Super Contributor
timestamp: April 28, 2021, 09:06:37 am
content: Quote from: 2N3055 on April 27, 2021, 07:02:01 am+1. That's why we need technical writers, managers, so on.

author: wizard69
author_rank: Super Contributor
timestamp: April 28, 2021, 02:49:08 pm
content: First let me say that I don't think much of the home automation in the way it is often presented. In fact I see it as a massive waste of money. So I hope the comments below don't seem to be too harsh.First; the complexity of these systems is mind boggling and frankly your first post sorta highlights this. I mean I have no idea what you have going on with this system and I wasn't too far into the posts that I drifted off. I just don't see the point in fiddling with such complexity in a system, I get that every day at work.Second; The distributed system you describe has me asking why? This seems to be the opposite of the KISS principle. A master controller and some slaves are far easier to understand.Third; If you are thinking a bit about ladder logic and PLC's you will have a good idea what might fly as a simplified solution for the average person.

author: EtaPhi
author_rank: Contributor
timestamp: April 28, 2021, 04:32:40 pm
content: QuoteThis is another my presentation fault.If you find uAntColony model mind boggling, academic world laught at it because it's obvious, it's a toy for them.Have you ever studied Kalman filter equations?https://en.wikipedia.org/wiki/Kalman_filterThat's academia want!I wrote that model because it's a proof that uAntColony concept works.So, even if you didn't build a prototype, that model grants that all is ok.QuoteSorry, a uAntColony is a simple and efficient home automation system (IMHO).You may have not understood (and that's my fault again) that uAntHill is a master and uAnts are slaves.There is only one active uAntHill which controls bus trafic, i.e. is a bus master.uAnts can send data only as a response to a uAntHill request, i.e. they are bus slaves.There are also some fail-safe features, such as when primary uAntHill fails, a secondary uAntHill may take over it.I did not write about these details to shorten presentation since I think that it's better to have a whole view of a system that to get lost in its details.QuoteI think that PLC and ladder programming are things too complex for an average housewife.Home automation is hard to sell to them.Today remote are sold as home automation parts.A smart socket that you can switch on or off by voice commands or by your smartphone is only an expensive remote controlled socket IMHO.When things become a little complex, such as a light can be switched on only if it's dark, or an air conditioner can work at full power only if PV system produces enough energy, those smart remotes seem not so smart...

author: phil from seattle
author_rank: Super Contributor
timestamp: April 28, 2021, 04:57:05 pm
content: Quote from: Vovk_Z on April 28, 2021, 09:06:37 amI'll take it one step farther - and a leader that sees the best end result for the customer (user/buyer/builder/whatever). Apple didn't get successful by exposing the inner workings to the end user.There was a great Honda ad from 20+ years ago. It showed a car with half the surface skin removed and showing the incredible engineering detail that went into the car. The tag line was "We make it simple". That message really stuck with me. If you can't make it simple for your target customer to use, don't bother.

author: Renate
author_rank: Super Contributor
timestamp: April 28, 2021, 05:36:38 pm
content: What we're all struggling with, is how is your concept different than a bog-standard SCADA system?

author: EtaPhi
author_rank: Contributor
timestamp: April 28, 2021, 08:25:01 pm
content: QuoteSimply put, it's like a SCADA who it's in control of all...A SCADA usually sends state enquire commands, while low level automation runs.PLCs answer to those state enquire commands while they execute low-level automation loop, i.e. in each cycle they acquire inputs, compute new outputs and actuate their new values.All works well if there are no remote inputs or if a fieldbus is reserved to PLCs, otherwise state enquire commands may interfere with normal PLCs operation by subtracting communication bandwith.A house is different from a factory. For instance it's even difficult to install a bus and reserved cabling is almost infeasible. When there is a bus, it's shared between all devices which must be small to fit in junction boxes or behind switches plates. These devices must be cheap too and have to live with existing devices because home automation does not happen immediately, but usually it takes time to replace old "dumb" devices.State based automation allows to add devices without changes to previous ones.Command based automation is less flexible, as the device that sends a command must know what commands accepts the device which will execute them.For this reason, when a new device is added, all devices that send commands to it must be modified.uAnts, on the contrary, have not to be modified, because their state doesn't change when a new uAnt is added.Only the output functions of the new uAnt have to be defined, which is reasonable.uAnts fail gracefully, because when a uAnt stops working, its state doesn't change anymore.uAnts that depend from its state will keep working, perhaps their performance will degrade, but nothing stops working as happen when a PLC fails.uAnts Protocol is efficient and makes it possible to trasmit as few bytes as possible.For instance, SCADA state enquire and setup commands aren't needed, because all communication is a state enquire (this is true 99% of time, as there may be state write sequences too).

author: james_s
author_rank: Super Contributor
timestamp: April 28, 2021, 08:43:14 pm
content: I still don't understand what any of this does for me. I add and remove devices to my automation system all the time without messing with existing devices. Commands vs state is irrelevant to me, the automation system knows the state of all of the devices, it changes the state by sending commands and monitors the state by receiving commands. How will some new system provide a tangible improvement over what is already available and in widespread use?

author: Bassman59
author_rank: Super Contributor
timestamp: April 28, 2021, 08:57:23 pm
content: Quote from: phil from seattle on April 28, 2021, 04:57:05 pmI think I see the disconnect here.What Phil from the PNW is saying is true -- it has to be simple for the end user. (The user is canonical housewife mentioned earlier.)But if you've been around stuff long enough, you realize that making things simple for the end user can require really complex engineering under the skin, like in the Honda ad.Think about USB. The user is trained to just plug it in, maybe install a driver, and then it Just Works. If you've done any USB device development, you know that it's quite complicated.EtaPhi describes the "under the skin" implementation. It needs a skin to hide the complexity. Users don't care about the implementation details. They care that it works and is easy to configure. Get a software developer with user-experience experience to come up with good interface and it could take off.

author: james_s
author_rank: Super Contributor
timestamp: April 28, 2021, 09:05:24 pm
content: Sounds reasonable. I don't really care what is going on under the skin as long as it works. It is far more important to me that it works with a variety of off the shelf hardware, I'm not going to build my own lamp interface when I can buy a Sonoff or other commodity smart socket, pair it to my system and go. We are long past the era of people building all of their own automation hardware, the market is flooded with good and inexpensive hardware, usually where it is lacking is in the "glue" that holds the system together and that is where the open source systems really shine. I have a mix of wifi, zigbee and RF devices and everything just works, it's almost seamless.

author: EtaPhi
author_rank: Contributor
timestamp: April 29, 2021, 07:14:31 am
content: QuoteThere is a small improvement from your point of view.You pay for having a supervisor system (even if it's free and open source) because you have to run this software h24 24/7 on hardware that draws energy.The yearly increase of electric energy bill is small, but there is.uAntHill is like your supervisor system.Your modem router may run its software because it needs only a UART to send and receive bytes.You obviously have to modify your modem router firmware, so it's not a task for a housewife until, some day (I hope) modem routers have it built-in.You may not agree with this example.I therefore tell you a thing that has been feasible since twelve years ago and that your system could not do: lamp configuration without any computer or smartphone or software.As I said regarding address assignment procedure, uAnts have got a button.After a uAnt has got its address, this button may be used to configure outputs activation.If you keep pressed this button long enough, uAnt enters in "output configuration" mode.You select the output to configure by turning the potentiometer which is usually used to lock uAnt configuration.Once an output is selected, you walk to the uAnt whose state must control that output and press shortly the button to put the latter uAnt in "stimulation mode".When a uAnt is in stimulation mode, a digital input change toggles pairing mode, i.e. a bit in uAnt state say "I control you" or "I don't control you".After a flashing led confirms that stimulation has been broadcasted, the "programming" button may be pushed again to resume normal operation.If there are more controlling uAnts, for instance when a lamp must be switched on/off by two or more points, you repeat the above simple steps.Finally you walk to the uAnt in "output configuration" mode and push the "programming" button.From then on, when you push one of the controlling buttons, the lamp is switched on or off."Under the skin", this is simply obtained by how state byte 0 is defined.This byte always encode 4 digital states as pairs of bit.Pair 00 means "I don't control you"Pair 01 means "I'm pushed"Pair 10 means "I'm not pushed"Pair 11 means "I control you"When a uAnt is in normal mode, it discards digital pairs 00 and 11, so the other uAnts which use that state bit are unaffected when a uAnt is in "stimulation mode".On the contrary, when a uAnt is in "output configuration" mode discards digital pairs 01 and 10. Since uAntHill says which address is now speaking, the uAnt in "output configuration" mode can manage the list of its controlling states.

author: james_s
author_rank: Super Contributor
timestamp: April 29, 2021, 09:00:15 pm
content: Quote from: EtaPhi on April 29, 2021, 07:14:31 amMy supervisor system is a Raspberry Pi that cost me under $60 with the SD card and power supply and consumes around 4 watts. This cost is negligible and is far more than offset by having a central supervisor that allows me to manage and configure the entire system all in one place. Having a supervisor system is not a bug, it's a feature, it's not a disadvantage, it's an advantage. It makes my life easier and saves me time, for a cost that is all but irrelevant. There is huge value in a central device that monitors and configures everything, logs data and serves as a gateway to the outside world. Sorry I'm just not seeing any advantage to some other architecture.

author: Fredderic
author_rank: Regular Contributor
timestamp: April 29, 2021, 09:58:24 pm
content: I actually went down a similar rabbit hole about 20 years ago… Designing a self-hosting distributed system not unlike this ant hill thing. In mine, a HUB was used to connect the devices network to any outside network you wanted (such as your PC), and to update the rules, including when the network topology was changed (was probably going to use good ol' RS-485, think I was also looking into CAN).The HUB would compile the rules, and then use the current network map to build a set of instructions for each node telling it what devices needed to know about it's states, and the network paths to send the update packets there (basically a binary sequence of port numbers… most devices had an in and out port, larger devices might have three or four ports, facilitating a star network topology, cycles allowed)… This allowed the nodes to be extremely simple, and handle message routing without having to actually know the topology, deal with discovery, and all the rest — the HUB would receive a message about a new or missing node, query it for info, and as needed, update the end nodes with new instructions.So long as you didn't need external connections, weren't moving nodes or network links around (including fall-back through alternative routes), you could remove the HUB entirely and it'd all just continue to work on it's own — notably, since the nodes spent most of their time in their deep sleep states and could readily fall back to batteries, I had planned to include a fall-back route support which only bothered with the battery-backed nodes to, for example, keep door locks functional as long as possible in the event of a power outage. The HUB would also monitor traffic, though, and adapt the network accordingly when alternate routes were available. I was kinda interested in message routing algorithms at the time, and it meant you could just throw as many links in as you wanted between nodes with sufficient spare ports, and let it sort them out, and maybe even suggest improvements.I had it mostly working and was just about ready to start moving it beyond bread-boarded prototypes, when I noticed all that consumer stuff was appearing, and decided we didn't need yet another standard, and no one really seemed to want wired any more, anyhow…One of many projects that in hindsight I wish I hadn't scrapped just because someone else was doing something similar too…BUT, these days, efficiency of messaging just isn't an issue any more. MQTT carrying JSON packets still gives my 8-bit brain the shudders, but, my current mixed WiFi and Ethernet Home Assistant based automation system manages just fine — and honestly, the UI isn't THAT bad…! Plus there's whole-screen cards these days that let you lay out whatever you want, however you want, and even mix-and match to a degree, pretty readily. I generally have a screen laid out for PC's (3 or 4 columns), a screen laid out for mobiles (single-column, I admit the layout algorithm is a bit naff, but, I hear they're working on it, and I generally like what I hear), and a screen where I let it just toss everything in together however it likes (catches those brand new things I've just plugged in and haven't gotten around to placing into the main views yet, or internal stuff that I really didn't think I'd even want to have exposed), and it really does work beautifully.I completely agree it's the way to go these days, and the vast variety of gear a big core system like HA can connect to is just amazing — by far way beyond anything I'd ever hoped to achieve. And in the few years I've been running it, it's only let me down once locally. Biggest issues are with the cloud connection which has hiccups on occasion… But there again, I never could have done the whole system, support for all the components I've got tied into it,anda pretty decent mobile app on top.Do I still wish I'd built out my system? Sure. I think it'd still be slightly better for controlling the lights and things like that over its wired connection. But is it worth doing it at this point? Not really, no. The appropriate compliance marks on the switch nodes I use instead aren't nearly as likely to make the insurance company run screaming if there's ever a problem. And quite frankly, I've had more failures due to bulbs dying, than I have with Home Assistant failing in any way — it's performed admirably, survived countless updates, and runs just fine on its R.Pi without so much as a heatsink (though, I probablyshouldactually stick on the heatsinks I have for it… some day…).

author: wizard69
author_rank: Super Contributor
timestamp: April 30, 2021, 02:18:19 am
content: Quote from: EtaPhi on April 28, 2021, 04:32:40 pmThe idea behind the PLC statement isn't to replicate a PLC but rather the concept or abstraction of an industrial control system. What you would need is something that abstracts you home control system into and easily operated graphical interfaceQuoteIt is hard to sell to anybody. I mean seriously what is the motivation here, I consider myself to be a normal male with a deep involvement in technology and I just don't see a benefit. The set back thermostat that I installed years ago and tore out last year, was more of a pain than it was worth. There was a huge challenge in keeping the batteries from leaking and frankly it ended up emulating a mechanical thermostat with a bunch of useless technology in the box.QuoteSold is not the same thing as grabbing a huge market.QuoteA PV integrated air conditioner might be a good thing but we still don't need a whole home automation system to accomplish it.Maybe I'm biased but I'm nearing retirement and frankly I'm thinking about moving. I don't need a huge home anymore and as such I'd likely move into a smaller home, with a open floor plan (as much as possible anyways). So I really don't see the point in home automation as long as I'm healthy enough to adjust the temperature when I'm cold. If I was to build such a home tomorrow I just don't see home automation as being very high at all on the list of must haves. Solar power would be much higher on the list as would be a workshop to fiddle in. In fact I probably could come up with a very long list of must haves before any sort of home automation lands on the list.Now I realize that not everybody feels this way but on the other hand I don't see a huge segment of the population running out to buy home automation hardware. There have been shots taken at the home automation concept for decades now and I think what these developers don't get is the WHY?

author: james_s
author_rank: Super Contributor
timestamp: April 30, 2021, 02:41:27 am
content: Quote from: wizard69 on April 30, 2021, 02:18:19 amFor me it's just a hobby, I guess I inherited my grandfather and my father's love of gadgets. My grandpa especially loved that stuff, he had every kitchen gizmo imaginable, electric carving knives, the first immersion blender I ever saw, soda maker, ice cream maker, sandwich maker, burger cooker, one of the very first Amana RadarRange microwave ovens, the first color TV my mother ever saw, I'm sure you get the point. Home automation wasn't much of a thing back then but he would have been all over it if it was.I do like being able to control lights from my phone or a wireless remote, my partner has a tendency to leave lights on and now I can turn them off without getting up from the sofa. I've also retired all of my electromechanical lamp timers and I use Home Assistant to schedule all that stuff. When I go on vacation I can keep the place looking occupied by setting lights to turn on and off at various times or just pull out my phone once in a while to switch them manually from wherever I am. One of the more useful bits of automation I set up is using the motion detection of my front porch IP camera to turn on the pathway lights if it detects motion while it's dark out and this works independently from the timed schedule that turns them on from sunset until 11pm. I put water leak sensors under all of the sinks and that has saved me from a big mess once so far when the dishwasher vent clogged and started dumping water into the cabinet under the sink, something that had happened in the past and caused some damage. I wired modified door/window sensors to my washer and dryer to send a notification to my phone when either of them finishes. None of this stuff is anything I *need*, but cheap off the shelf hackable hardware has meant I have not had to invest very much money and I've been having fun playing with it.It's also worth mentioning that the Tasmota firmware I use on most of my esp8266 based devices is capable of working standalone without a host supervisor. It has built in timers, control through a web page hosted by the device itself and they can send and receive mqtt messages so it's possible to set up a standalone peer to peer system, useful for very small installations where maybe you just want to control some lights with motion sensors and timers or something. Overall it isn't something I'd recommend to non-technical folks though, if I didn't enjoy tinkering I would not have bothered.

author: EtaPhi
author_rank: Contributor
timestamp: April 30, 2021, 12:44:49 pm
content: I thank all who posted.Their replies let me realize that there is nothing wrong with the technology of my home automation project.People however don't care of technology as long as their needs are satisfied.Home automation concept is different from people to people.Automation experts see it alike other automation systems they deal with daily, i.e. a set of PLC and SCADA connected by some field bus.Technophiles see it as something they can control with their smartphones or voice.I see no intelligence in the latter view, as that home automation has not to be smart, but execute what it's orderer them to do.Even thermostats, which should be a bit autonomous, are a pain to manage.It doesn't matter if a 32 bit or 64 bit computer does what a humble 8 bit microcontroller did more than a decade ago for a tenth of the cost using only 8192 bytes of code and 384 bytes of RAM, as these computer are cheap.I don't want to sell anything, as I wrote in the preliminary remarks.I wanted to share an idea which influenced my following projects, i.e. a photovoltaic system supervisor and anti theft system, a drone control system where aviation safety standards are followed (but certification has to be started, because it's still an unfinished hobby project).This latter project uses the state propagation concept too.There are however two CAN buses as one may fail and there are safety measures which halt a misworking unit if it sends CAN messages when it's not allowed to speak.Different requirements, different solutions.I wondered if my view of automation was biased, if uAntColony and state based automation was a dead end experience.It wasn't, as what really counts are results, not how you achieve them.Thanks again to all