Creating a communication protocol between devices - Page 1
Forum section: electronics / projects

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 09, 2020, 05:16:46 pm
content: Been brainstorming for a while a project I want to start on. It's not too complicated and I'm sure there are many like it but I still want to make my own. Essentially it will be a home automation/monitoring system where you can have various sensors/controllers throughout the house, or even across multiple buildings. My goal is to make it very plug and play, so you would start with the master which would be a PC, it can be a raspberry pi, basically anything that runs Linux. It would run software that then talks to the central hub which talks to all the sensors. Then you would have the central hub which would be essentially an arduino/MCU board connected via USB (serial interface) to the PC. The central hub would then have various sensors, or even other hubs connected to it. This is where the protocol comes in. My goal is to make this super plug and play which is why I want a custom protocol. You plug in any device you can think of that speaks this protocol, and it would just show up as a new device and then you can configure it and do what you want from there. A device can be as simple as a dry contact i/o board, or a single temp sensor, or another hub, or even something super specific like a custom hvac unit or something (actuators, sensors etc to control/read). Goal is to make it super expandable. You add a device and it would essentially present input and output and other similar parameters. It would be up to the central controller (on the PC) to decide what to do with those controls.So I guess my question is, what is a good reading resource on designing a protocol like this? Essentially communicating between other MCUs over potentially long distances. (let's say, up to 1km but probably much less) I want to read up on stuff like different ways of doing it, such as doing a single wire bus, or a two wire bus where you have a clock signal, twisted pair vs not, etc... There's many ways to go about it I'm sure.I'm leaning towards a 4 wire setup. You have your ground, +5v, a clock signal, and then the com wire, which would use 0/5v to speak in binary. The actual devices would use 2 pins for this, an input and an output, with appropriate diodes etc to ensure they don't screw with each other. Only two devices would be able to speak at once so it would use a sort of polling system. The clock would only go on when it's that device's turn to speak. This is very simple rough idea mind you. Basically open to suggestions on what to read up on while designing this. I'm sure there's lot of pros/cons I need to consider on various ways of doing it, such as how far I can go or how many wires I use up.I would probably lean towards atmega just because I have lot of those chips, I only really know how to do arduino at this point but I would probably want to learn how to do it more "raw" to save on memory space. On the other hand arduino has lot of libraries so I can google "how to do x in arduino" and get way more results than "how to do x in atmega" for example. But once I find info on how to do it on arduino I can still translate it as needed.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 09, 2020, 05:37:38 pm
content: This is almost exactly what I've been thinking of too.First obstacle I found is that it seems that it wouldn't be possible to have a single capable PnP protocol both on PnP/discover/routing level and at end device level because of memory requirements for end device controllers.For upper level between main PC and hubs I'm almost convinced about an Ethernet-like network over cheap RS485/422 physical level and INDI-like (https://indilib.org/) protocol for reading and writing data. For lower level between hubs and end devices I'm looking forward to something like MODBUS with custom packet payload format or something else lightweight. For both levels each device/hub/etc must have a hardware UID for MAC, 1-wire or MCU built-in.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: November 09, 2020, 05:39:39 pm
content: Sounds a bit like CANBUS?https://en.m.wikipedia.org/wiki/CAN_bus

author: Unixon
author_rank: Frequent Contributor
timestamp: November 09, 2020, 05:52:53 pm
content: Quote from: Syntax Error on November 09, 2020, 05:39:39 pmI've dropped CAN after long hesitation. Small payload per packet, relatively expensive transceivers, short fixed length UIDs, overall rigidness.

author: pwlps
author_rank: Frequent Contributor
timestamp: November 09, 2020, 06:15:58 pm
content: Quote from: Red Squirrel on November 09, 2020, 05:16:46 pmI'm not sure I understand, this would be a half-duplex link ? I think it will be easier to design a protocol if you opt for full-duplex like in twisted-pair ethernet. The number of wires shouldn't be a limiting factor as 10BASE-T cables are cheap and provide more wires than you need (4 pairs). Also if you want to go up to 1km length you don't have so much choice for the cable selection, differential signalling and well controlled impedance will become important as length increases.

author: DrG
author_rank: Super Contributor
timestamp: November 09, 2020, 07:05:45 pm
content: My goal is to make it very plug and play, so you would start with the master which would be a PC, it can be a raspberry pi, basically anything that runs Linux. It would run software that then talks to the central hub which talks to all the sensors. Then you would have the central hub which would be essentially an arduino/MCU board connected via USB (serial interface) to the PC. The central hub would then have various sensors, or even other hubs connected to it. This is where the protocol comes in.For consideration:ESP8266 or ESP32 all running servers on the sensor ends.Main controller (also an ESP32 or ESP8266) is a client that queries each of the servers.You can just communicate using plain socket communications. You don't even need to use HTTP, although you could if you desired the punishment.Note this can be entirely local and you can add additional security. Your router and ISP are not needed here.Sure, you need to think about transmission distance, but hey, no wires. Within the communications packet, you can add your standard communication information to customize the packets.There are dozens of examples of this on the web.

author: ogden
author_rank: Super Contributor
timestamp: November 09, 2020, 07:31:07 pm
content: Very plug and play? - ZigBee.Of course, there are loads "cable guy" stuff like LonWorks, KNX, Dali to name a few. I don't think there is need for next buildingautomationwired interface.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: November 09, 2020, 09:01:39 pm
content: A general question. What layers on the conceptual OSI model is your protocol covering? You're building for the physical layers, but how do your devices plug and play with each other? You would need to specify the protocol at the higher session and application layers too.Open Systems Interconnection model (OSI model) wiki:https://en.m.wikipedia.org/wiki/OSI_model

author: max_torque
author_rank: Super Contributor
timestamp: November 09, 2020, 09:41:23 pm
content: Most important question: COST???An ethernet based solution is going to bring the greatest capability, with fast, high bandwidth exchange and the possibility for nodes to include a lot of processing power.An basic two wire serial (RS-485 etc) is going to be the cheapest, but with a much lower bandwidth and "dumber" nodesSo, which is more important, cost or performance?If you are unlike to be exchanging a lot of data in a short time, ie the networks is mostly going to be exhanging 10's of and up to hundreds of bytes rather than thousands or millions, then the advantages of the basic serial approach are difficult to ignore. And basic serial using simple micro's with UARTS and your own protocol mean you can make the transfer as simple or as complex as you want.A std packet with byte stuffing and a simple check sum would allow you to exchange from say 1 to 100 bytes in each packet, and you could run a fairly low baud to keep the link reliable over longer distances. If you use RJ plugs and std ethernet cables (cheap, easily available, relatively robust) then you could have a full duplex RS-485 link and probably power all in one.Talking of power, this is probably your limiting factor in terms of the bus architeture!

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 09, 2020, 11:48:08 pm
content: I'm looking at layer 1 at the moment. And CHEAP. Something that won't require any special transceivers etc, something I can just big bang with regular I/O and add any supporting jelly bean parts as required for things like input protection etc. That's kinda the info I'm seeking as well since I'm not too sure about those parts. I was actually considering doing ethernet so that standard equipment like switches etc would work to add more ports but it's much more involved than I want to get into.Essentially once I have layer 1 decided I can start thinking about the more specifics. An end device may very well use i2c, canbus, or whatever is supported by the particular chip being used (ex: humidity sensor), but to communicate with my custom network it would use my own protocol. I feel it just makes for a more unified experience that way as every device works the same across the board. I may also incorporate zigbee and such if I want wireless, but my protocol would probably just ride on top of it. Ex: there would be a wireless "sender" and "receiver" then it just translates my protocol over zigbee. When you plug it in, it won't even know the difference as the transceivers would do all the work but it would just act like a wired device.Regular ethernet will also be used to some degree for long distance networks, as there could be multiple "RTUs" that are part of the overall system. But that will just ride on the normal IP network and it will be the responsibility of the actual Linux host to handle all that. Ex: the remote RTU would just be another RPI running the same setup, but the master one is just talking to it. It will basically work like a large hierarchy network.The data rate is going to be fairly low, like maybe a couple KB/sec and that's probably even pushing it. Though there's a possibility I look at higher data rates if I want to be able to control more sophisticated things like say, an array of LEDs or something. The system would work in a polling fashion where each device is polled individually, so I don't plan it to be super fast.

author: S. Petrukhin
author_rank: Super Contributor
timestamp: November 10, 2020, 01:31:54 am
content: Initializing the exchange from the server (PC) is not always convenient. This causes all endpoints to stay awake waiting for a request. Some sensors, such as thermometers, do not make sense to poll often, they can be powered by a battery and sleep most of the time, periodically wake up, send information in a short period of time and fall asleep again.First, you need to determine the amount of data to be transmitted, the appropriate physical layer and transmission medium for this volume. You can't rely on streaming video, for example, over the RS-485 bus.And already using ready-made systems, for example, MQTT or making your own top-level protocol is the easiest question.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 10, 2020, 01:40:59 am
content: Actually yeah that's a good point maybe going with a polling architecture is not the best way, as if I do incorporate wireless stuff I would want sensors that can go to sleep. Though the part that is physically connected can always still stay awake and when it gets polled it would just send the last value it got from the wireless portion.I could do a bus where everything is just connected and talks when it wants, but I feel this is more complicated from a software point of view as you need to make sure not more than two devices speak at once. Though this is the sort of thing I'm looking for on what to read up on. Various techniques to have multiple devices talk to each other and best way to go about it. Whether I use a bus, or other method.

author: S. Petrukhin
author_rank: Super Contributor
timestamp: November 10, 2020, 01:58:12 am
content: Quote from: Red Squirrel on November 10, 2020, 01:40:59 amThe issue of competition for the transmission medium has long been resolved by science. If you use radio communication, then high WiFi protocols, LoRa will solve the issue of competition for you, if you use ethernet, even in the form of a bus, then you will not need to worry about competition. By the way, using the WZ5500 you can very easily organize TCP / IP easier than ever. A lot of different options, but even TokenRing without fighting for the environment, even FrameRelay.Buses such as RS-485/432 require care for traffic management. If you are planning to make some kind of home network, then one of the options is to use an IR environment that implements the principle of ethernet competition, for example.

author: S. Petrukhin
author_rank: Super Contributor
timestamp: November 10, 2020, 02:00:21 am
content: Quote from: Red Squirrel on November 10, 2020, 01:40:59 amLearn how ethernet works at the physical level, how it deals with collisions - a very simple solution.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 10, 2020, 08:33:26 am
content: Quote from: max_torque on November 09, 2020, 09:41:23 pmPower distribution must be through a separate network IMO, powering from the bus works only for isolated bus transceivers and low power end devices.Another important point is bus voltage and max current.Bus voltage must be higher than end device working voltage to allow local filtering and precise power regulation with inexpensive linear solutions.However, putting a voltage on a bus outside of signal voltage range is dangerous, if cable is poorly crimped or damaged this failure may have a high cost.Low bus voltage is safe but losses are higher and this may require step-up converters at each end device.A typical twisted pair is capable of carrying few hundreds of mA of current (200-500mA AFAIR).Cross-powering through a bus is bad idea, balancing multiple local regulators is a nightmare and without balancing it's a failure in the first place.Power must be distributed only from top to bottom and not the other way around.Hub to hub connection must be isolated just because there is high probability that hub will be powered from different power sources with high voltage difference between them. Hub to end device may be or may be not isolated, that depends on a particular case.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 10, 2020, 05:47:47 pm
content: Yeah isolation is something I want to keep in mind too. Either make it part of the design so each device is isolated, or only isolate the things that really do need it (Ex: voltage sensors or devices that have a secondary power source). I do want to keep power as part of the bus though, as it minimizes the need to provide external power to a sensor (when I say sensor here it can be an output device too like a relay), and also reduces the need to have many backup sources. Though perhaps sending only 5v is not enough, and perhaps I can do 24 or even 48v then the isolation transformer just has a tap to bring it down to 5, but also has an isolated 48v to send to other sensors if desired. (ex: for a hub)One approach that's crossed my mind, but would be a bit more expensive per sensor is to make it so I have a universal board that sensors plug into. The board would be at the point of use and handle the isolation, step up/down of voltages etc. The sensor would just be another board that plugs in and gets it's power and I/O pins. The nice part about that though is you can quite easily swap out a sensor for any other type if you really want. Like say you have a temp sensor and want one that does humidity too, you would just swap them out and the system would lose the old one and the new one would show up. Nice for a servicing point of view too. Some sensors would be fairly inexpensive while some would be more expensive so being able to switch just the sensor and keep the magnetics (power isolation xformer, opto couplers etc) intact is a bonus. Or if magnetics part fails you swap that out and the sensor itself is still good.Eventually I do want to get into more complex stuff like boards that can run a real time OS, at that point I would just use ethernet, but for now this is going to be a fairly dumb and basic network.

author: pwlps
author_rank: Frequent Contributor
timestamp: November 10, 2020, 06:28:54 pm
content: Quote from: Red Squirrel on November 09, 2020, 11:48:08 pmI don't have any specific advice, it's just to define the possible options and to emphasize on differences between them:1) fixed clock frequency for both sides like in rs232 and ethernet, the receiver has to recover the clock phase from the signal,2) master-slave configuration where the clock is provided by the master side, like in SPI,3) a fully asynchronous scheme like gpib where there is no clock signalEach scheme has its own advantages and disadvantages/limits.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 10, 2020, 07:23:30 pm
content: Quote from: Red Squirrel on November 10, 2020, 05:47:47 pmThis approach partially solves hardware part of the problem but not completely.This gateway device must be somehow wired to an actual sensor/actuator/etc or a MCU that controls sensor/actuator/etc. If there's no MCU in an end device that would contain a software driver for an actual sensor/actuator IC in its firmware, the gateway device itself must have a software driver for that sensor/actuator/etc and hence it's either not universal or must have a bloated firmware that knows how to talk to every piece of hardware and IC out there.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 11, 2020, 07:02:42 am
content: Yeah with this approach there would basically be two MCUs involved, a standard one that has the same programming for each one and handles the isolation etc (somewhat acting as a power management device too as there would be a transformer involved). This portion could perhaps also handle addressing and other administrative stuff, so the sensor itself has a bit less repetitious code. Then the sensor board that connects would have it's own MCU with programming that is specific to the onboard sensor/control.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 11, 2020, 01:14:28 pm
content: My point was that this way there will be one more protocol in the system, between gateway MCU and end device MCU with specific firmware.The purpose of the gateway in terms of software is to offload network operation from device-specific MCU thus making its firmware smaller or just not bloating it beyond available memory, and this gateway MCU has to talk to end device MCU in an uniform way across different end devices.And the question of what this protocol may be is huge one.End device MCU may be as tiny as 1k AVR or as large as 1M C-M3/M4 ARM for example.Smallest of them won't do anything above very simple IO, and the largest of them may be capable of handling network functions all by itself.

author: ogden
author_rank: Super Contributor
timestamp: November 11, 2020, 05:52:17 pm
content: Quote from: max_torque on November 09, 2020, 09:41:23 pmEven more important is actual ability to install wired system. Many houses and flats are not compatible with easy/cheap retrofit of extra wiring - no matter it is Ethernet, RS485 or not. Instead of reinventing technology of last century, you may consider wireless (wifi/lora/zigbee - pick your poison) . There are many examples, first search hit as one:https://www.aqara.com/us/products.html

author: Unixon
author_rank: Frequent Contributor
timestamp: November 11, 2020, 06:21:40 pm
content: Quote from: ogden on November 11, 2020, 05:52:17 pmThat's a completely different story, they won't be able to use any good stuff anyway, only consumer level wireless or locally wired gadgets. Forget it.Wireless systems are less secure, they both produce noise and susceptible to jamming and interference.And most disappointing thing about wireless systems when it comes to automation: no power distribution.However, wireless bridges may be considered useful parts of the communication system.

author: ogden
author_rank: Super Contributor
timestamp: November 11, 2020, 07:52:54 pm
content: Quote from: Unixon on November 11, 2020, 06:21:40 pmSecurity through obscurity or physical access is way less secure than encrypted wireless. Anyway seems you need to update your knoledge about encrypted wireless and how secure it actually is. If you think that somebody is going to jam your wireless home automation system then you would want to reconsider automation in your home as such.QuoteNonsense argument. What you need power distribution for? In case of sensor - use battery. Otherwise you are getting power anyway.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 11, 2020, 08:39:25 pm
content: Quote from: ogden on November 11, 2020, 07:52:54 pmI agree about obscurity part, but disagree about physical access.You can't challenge a system you have no knowledge of its mere existence and cannot discover it.Wired channels can be as well encryption protected as wireless channels.Quote from: ogden on November 11, 2020, 07:52:54 pmToo many wireless modules on small area in the same band can perfectly interfere with each other operation thus disrupting a network.Quote from: ogden on November 11, 2020, 07:52:54 pmWorks for single gadget, doesn't work for many.The logic behind this is following: if you have an electronic device it requires power => if you provide power through wires, you have wires => and if you have wires, you have place to run them to and from that device => if you have a place to run wires, you can also put there data lines.

author: ogden
author_rank: Super Contributor
timestamp: November 11, 2020, 10:03:34 pm
content: Quote from: Unixon on November 11, 2020, 08:39:25 pmSorry to disappoint you - wired communication systems can be discovered as easily as wireless unless insanely shielded.QuoteSo what's the advantage of wired systems then?QuoteUtter BS. You really shall update your knowledge of communication systems. Start with keywords: CSMA/CA, CSMA/CD.QuoteWhen you have mains power somewhere it does not mean that you automagically can have Ethernet or other kind of wires there, kid.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 12, 2020, 01:36:07 am
content: Quote from: ogden on November 11, 2020, 10:03:34 pmThis kind of discovery requires much closer access than systems that broadcast themselves over the air to entire neighborhood.It doesn't even matter if a particular communication system is wired or not, technically its the question of detection thresholds.The point is - in order to go searching for a communication system one has to have at least an idea that something is there.And until you raise what you search for from noise floor by any method you can't tell if it exists or not.Quote from: ogden on November 11, 2020, 10:03:34 pmThey are simple and straightforward if you're not overcomplicating them intentionally.Quote from: ogden on November 11, 2020, 10:03:34 pmOh, I know. Just keep increasing number of agents until it gets too overcrowded and fails or agents start indicating timeout while waiting for a free slot.Quote from: ogden on November 11, 2020, 10:03:34 pmThis depends on how the wiring is done and what are the rules for electrical systems.In older places you can't do anything about wiring if it is mounted in concrete walls behind some decoration on top of that,in new like office places all wiring is in channels with outlets and technical ceilings/floors and you can have almost anything there.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 12, 2020, 06:07:32 am
content: I do want to stick to wired, but adding a wireless bridge option as a type of "sensor" would be something I would eventually want too. Probably do zigbee for that part or whatever else comes up in my search at that point. But the main advantage of wired is power distribution and overall simplicity, as the goal of the system is to be very low maintenance and simple. Power would come from a central source that is itself battery backed up. Probably a 12v battery in a distribution cabinet that would power the whole thing. Kind of like a fire panel. Heck this system will be so modular it could technically be used as a fire system (maybe not officially as it would not have the proper certification etc for that purpose). Security is not a HUGE consideration with this, but I may very well eventually look into what it would take to encrypt data and have other measures in place to prevent tampering, but that's more of an aside for now. If I do add security then this system would be suitable for home security type sensors as well.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 12, 2020, 03:49:00 pm
content: Quote from: ogden on November 11, 2020, 07:52:54 pmLet me add few more things to the argument.I'm not saying that a wired communication system has overall better security than a properly encrypted wireless system just because it is not wireless.A system security is eventually measured by amount of obstacles for an adversary to get in and affect its operation and it is not right thing to dismiss inability to discover and connect to a wired system as valuable obstacles.We use encryption as the ultimate measure because it allows us to create huge obstacles with little effort.What I'm saying is adding protection by inaccessibility on top of encryption should not be neglected and that for communication systems that do not require high security the very inability to discover and/or access a system may serve as sufficient obstacle by itself.

author: nctnico
author_rank: Super Contributor
timestamp: November 12, 2020, 04:19:16 pm
content: Quote from: Unixon on November 12, 2020, 03:49:00 pmBut encryption isn't enough. Good security also needs a way to authenticate devices and a way to detect intrusions.But coming back to a protocol: there are 2 things to seperate:1) the physical layerAs a physical layer RS485 is good for long distances. You can use a UART or bit-banged Manchester code if you like. CAN bus is also be a good candidate because it also includes error detection and automatic re-transmission for un-acknowledged packets.2) the protocol itselfIn my experience readable text based protocols are by far the easiest. Look at why SCPI is so popular for test equipment; it is easy to work with, easy to prototype and easy to debug. The physical layer underneath it doesn't matter much.

author: max_torque
author_rank: Super Contributor
timestamp: November 12, 2020, 08:02:22 pm
content: I'ms till going to suggest that power distribution on your network is actually more critical than signal distribution!If you used say RS-485, then you could do cheap isolation using digital isolators to carry the TX, TX-EN and RX signals to standard (cehap) RS-485 transcievers. The amount of power each module requires is fairly small, so could be sent over the network from a master unit. The devices then are powered by local sources, or a dedicated "high power" distribution network, so seperating signals and power.One thing you need to consider with RS-485 / CAN is the fact that the bus needs to be linear and not in a star format, and how you can arrange the bus terminators. This is usually done by having each module daisy chain the bus through itself, ie there is an IN and an OUT plug, and the last modulei the linear chain has a terminator plugged into it's "OUT" socket.A basic protocol and physical layer allows for very simple and cheap slave devices. It also allows for very simple software too, which should not be under estimated. A basic 8bit processor with a std uart could be used for the simple tasks, and a more complex, expensive ARM for the complex stuff. ie keep simple, simple!if you are not going to send a clock signal as a physical signal, then your slaves will need suitably controlled clocking, or you will have to write some sort of power on "sync" functionality (not that hard) to recover the clock from say an initialisation string of data of known format sent by the bus master after it powers the bus up.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 12, 2020, 08:13:22 pm
content: Small. but important notice. Although RS485/422 is a bus, I would turn it into a set of P2P links just like Ethernet.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 13, 2020, 09:29:04 am
content: I kind of like the linear in/out approach where it's just a long string of nodes. It could be done in a very simple and cheap way. The way I'm thinking it could work is something like this:PC interface card - node A - node B - node C - etc... (technically as many nodes as I want) Have not decided how a terminator would work yet, I'm thinking it can loop back and plug right into the PC interface card. When the card gets a signal from there it knows it's at the end.The PC interface card would be very similar to a node, except it would have a USB uart and the PC talks to it via serial that way, and it talks to the first node. The nodes would then have their turn to talk to it's associated sensor that plugs in to the node board, then move on to next node. The PC interface card starts by generating a clock signal to the first node, indicating to the first node that it's time to talk. It would talk over the bus wire which would be common throughout the whole string. Once it's done talking it would turn off the clock, and start generating it's own clock towards the second node then that node would talk and so on... When it gets to the end it then just repeats the process.The actual sensor devices would then plug into a node, and have their own communication that independent of the main node loop. The node would keep a list of values in memory and it would simply update them with the sensor as needed. The nice thing about this is sensors that are slow to read/write would not hold up the whole bus, as the node would always be ready to send or receive values. For example if I have something like a linear actuator it takes some time for it to mechanically move and then read to confirm it's at the right spot. This would happen independent of the bus.From the PC's point of view the commands would be something like this:">" indicates PC talking,"<" indicates PC interface board talking to PC> get name, or some other form of start command< send name of sensor (this can be just a number such as the order in the string, or there could be ways to program that in)> get values< show list of values> write values (probably one command per variable)< Get an ok or other form of acknoledgement(repeat any other commands as needed)> Next node - This would simply move on to the next node in line and then the above commands can be repeatedThe values would probably just be written into a common location like a text file on a ram drive or something.A separate piece of software acting as an agent would then read/write to this and talk with the server. There could be several instances of these agents spread over a large network all talking to the same network and having multiple strings of sensors.The actual server software would then handle all the logic of what to do with these values. It's then up to the admin to program them to do what they want.At least that's the jist of what I have in mind anyway... the specifics I will of course figure out as I actually start.But yeah I'm actually liking this idea of a linear approach. Probably still do 4 wire, ground, power, clock, communication. (half duplex over same wire) Physically there are different approaches too, as I can have multiple nodes on same PCB and have it stay in the cabinet and the sensors then connect and go out to where they need to. Or I can have the nodes separate and just string throughout the building. If I use 4 wires it also allows to use cat6 and I can have it return back on the other 4 wires as needed to go back to cabinet. Could also use a hybrid approach where I have node boards with multiple nodes in each room/area that requires sensors. So you have cat6 going to multiple general areas of the building, then you can plug the sensors into it's respective board that would be in it's own small cabinet. You can of course just daisy chain as many of these boards as you want. They could come in different variants like single, or 4 or 8 etc... port.For power I'm thinking I could go with a high enough voltage like 24v or 48v. Having a higher voltage available would ensure having enough power for sensors that require a certain amount of power to operate (ex: some types of sensors like hydrogen sensors etc actually have a small heating element) The node board would do all the required power conversion stuff as well as isolation. Here I would be tempted to do something like have +5v but also keep +48v, the sensor would get both. That way sensors that may need a more specific voltage can have their own conversion stage and run off the 48v part.I'm thinking the nodes themselves would not be electrically isolated, but the sensors would be individually isolated from the node, thus, the entire bus. The PC interface board could also provide any input protection in case anything happens to the bus like high voltage accidentally applied to it. Or static electricity etc.At least that's one approach I am thinking.

author: max_torque
author_rank: Super Contributor
timestamp: November 13, 2020, 08:31:09 pm
content: I my experience it's easier to come up with a method of isolating the bus and power supply, meaning all modules of whatever type can use the same architecuture. And isolation at the bus level is most likely to be both a small number of signals (3 for RS4-85 half duplex) and they will be fully digital by this point, no matter what your sensor does. You slave logic then sits on the sensor side of that isolation, giving it free access to those sensors!The biggets descision will be how to arbitrate the bus and avoid collisions, especially with a half duplex bus. The simplest way is to only allow the master to initate coms, and all slaves only reply to that master when called on. No slave module can simple broadcast on the bus.A second approach is for the master to broadcast bus sync markers, ie certain characters to which the modules can sync, and hence wait to send out their data until "their" slot has arrived. That bus sync string sent by the master can also be used for auto baud rate determination and for baudrate clock syncing. That would probably allow you to run with slaves that use basic, low accuracy (and high drift, esp. tempco) microcontrollers without external xtals, saving costs and board real estate for the simplest slaves

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 15, 2020, 07:04:51 am
content: Yeah if I do the isolation at the node I would want to make sure that the sensor side does not really need to do anything special having to do with the isolation part, try to put as much of the work as possible on the node side, as once it's designed they are all the same and can be mass produced the same way. The hardest part will be the transformer as that is a bit harder to automate, but if I actually got to a point where I was making 100's or even 1,000's of these I would figure that out at that point. Or I can have two node versions, an isolated and non isolated one, as the isolation is only really needed for sensors that may connect to other voltage sources. (voltage sensors, or devices that will have their own external power)For arbitration, I realized that my idea of having the clock signal act as a token that goes along won't exactly work... the signal still needs to be generated by the master as it needs to be able to talk to the right node. Just an oversight brain fart while I was brainstorming there. So yeah the clock signal and bus need to be common and active the entire length. So what I need to do is as part of the "start" command to talk to a sensor, I need to have the master (that would be the PC interface board) send out an address it wants to talk to. Each node listens for the address when it sees a start command and if it matches it's own, then knows it's it's turn to talk and will issue a command to acknowledge it. I then just need to come up with a way to set the address and make sure it is unique. I have a couple ideas in mind for that, either it happens manually at time of install, or it can be part of a startup sequence when a brand new node is introduced into the network (ex: it could have some kind of initialization address). given each node would have a MCU I have lot of options for jumpers/buttons etc if I want to have some processes be initiated manually too, like a reset button.But yeah I've pretty much just been using this to brainstorm but thanks for all the things that have been brought up as it's all going to be part of the consideration as I start to actually experiment with this.I've never actually done anything to do with MCUs that involves having multiple talk to each other so I will want to experiment with that first at a super low level and then work from there. I'm sure I'll run into stuff that I forgot to consider as well, such as error correction, or how to detect a cut cable in middle of communication, etc.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 08:08:25 am
content: Quote from: Unixon on November 12, 2020, 08:13:22 pmI would not. - Because then adding new sensor means running new wire to the control hub. If system has to be DIY, easy plug and play then I would suggest to use ESP8266 as WiFi<->RS485(or PLC) gateways/end-nodes. I would use DC power line communication (PLC). System can have as many WiFi gateways as needed. Most likely my home controller would be rPI configured as wifi access point that serves whole home automation WiFI network - gateways, wifi sensors if any, user connections. Yes, susch system could be subject of WiFi DOS attacks any script-kiddie can do.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 09:23:07 am
content: Quote from: ogden on November 15, 2020, 08:08:25 amYes, it is. The logic behind that is simple: an end device and its bus connection may be physically hard to reach and if that device malfunctions due to hardware or software problems this can halt all bus operation and you won't know which device did this until you disconnect it and reconnect this bus segment to a bypass. With P2P links this is just one unresponsive link and more importantly you know exactly what link/device that is.Quote from: ogden on November 15, 2020, 08:08:25 amIsn't it expensive? I don't know, just asking. Mains AC modems are expensive. What parts would you suggest for this kind of circuit?

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 10:19:52 am
content: Quote from: Unixon on November 15, 2020, 09:23:07 amYes indeed there are such risks. Thou what are the chances? I am not saying that one shall put whole building on single bus - I said "system can have as many WiFi gateways as needed". If you expect hardware or software problems in your DIY building automation, then I am not sure you shall be doing such projectQuoteSorry, I did mean "PLC" in a broad sense, not AC mains only. What I had in mind is simple enough - DC bus with "voltage dip pulse" communication, likeKNX TP1:

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 11:45:17 am
content: Quote from: ogden on November 15, 2020, 10:19:52 amThis is a whole source of problems right there if you're building a system without thinking about things that may happen to it.No matter how good you are at engineering, no matter how high the quality of components and overall build quality is, you must suspect that some shit may happen.Whatever happens, a failed part of system must go down gracefully without causing any harm and with minimal effect on the rest of devices.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 12:32:59 pm
content: Quote from: Unixon on November 15, 2020, 11:45:17 amIf you are building system without thinking then choice of best network topology is least of your problems. There are many ways for single point of failure to bring down whole star network as well. Also when you actually think while designing common bus devices - signalling mode & interface circuit can be designed in such a way that failed device cannot bring whole segment down. What remains - disconnect or short. So it depends.Let's look at this from different angle. There are KNX TP1bus, Dalibus, Lonworks TPbusto name few. Do you really think that industry experts did not know what they are doing and you know better?

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 01:41:01 pm
content: Quote from: ogden on November 15, 2020, 12:32:59 pmThey were solving different problem with different boundary conditions and came to bus topology as an optimal solution in their cases.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 02:47:37 pm
content: Quote from: Unixon on November 15, 2020, 01:41:01 pmGood. There are no such thing as "their cases". They were building as universal systems as they can, that supports all network topologies one may need - linear, ring, tree, mesh, star and ... (surprise!) P2P. Proposed by you P2P approach is limited to only one topology, P2P. That BTW requires quite complex hub because every node needs individual communication port. You perhaps want star in your house, butothers will be solving different problems with different boundary conditions, will need tree or just linear bus.

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 05:16:21 pm
content: Quote from: ogden on November 15, 2020, 02:47:37 pmNo, I have to disagree. We all have our opinions on how things should work and what requirements they have to meet.Red Squirrel started this thread with his ideas, I dropped in few of my ideas, and so on... and none of these has to coincide with what KNX, Dali and Lonworks had on their minds while developing their products.Quote from: ogden on November 15, 2020, 02:47:37 pmIn fact, a network structure almost repeats physical location of modules, so it may have different topologies in different segments if physical interface allows that.To put it simply, I would like to see a simplified version of 100BASE-T Ethernet with cheap physical layer components, smaller footprint, easy way to make many port solution.For example, some of STM32F09x have 8 hardware UART interfaces in one package, 8-port RS485 hub can be compact and simple and do other functions besides communication.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 06:17:00 pm
content: Quote from: Unixon on November 15, 2020, 05:16:21 pmInstead of having unfounded opinion you could actually read standards to see that requirements and targets are provided.QuoteRed Squirrel can take any idea including mine or yours. Just please do not cry that I have arguments against your idea.  Actually Red Squirrel's targets: "super plug and play", "super expandable" do coincide with all mentioned standards. Your P2P idea - not that much.QuoteIn your "P2P" case every branch needs active multiport router. It increases cost and complexity, also not "very plug and play".

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 07:08:49 pm
content: Quote from: ogden on November 15, 2020, 06:17:00 pmOh, I will read them, but I highly doubt any of them fits perfectly.Quote from: ogden on November 15, 2020, 06:17:00 pmI don't know what was initially meant by this description of features, but that's how I see it:- Any device must be uniquely identifiable with hardware ID, typically 32, 48 or 64bit, could be more, whatever provided;- Any device must be addressable by its hardware ID using OSI Level 2 protocol;- Any device must discoverable by any other device using OSI Level 2 protocol;- Any device with 2 and more ports can be a hub [this may be mandatory];- Hubs are responsible for automatically discovering and updating network topology and routing data between any discovered UIDs on the network;- Any end device must be able to describe itself and provide access to its properties via OSI Level 3 protocol;- It is highly desired that Level 3 protocol would be human-readable, e.g. XML/JSON-based;- Any properties of any device must be remotely discoverable and accessible using OSI Level 3 protocol;- End devices must not require configuration prior to attaching them to the network;Maybe I missed some things, will update in later discussion.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 07:31:32 pm
content: Quote from: Unixon on November 15, 2020, 07:08:49 pmYou miss absolutely everything. I offered to implement"DC bus with "voltage dip pulse" communication, like KNX TP1"-physical layerfor bus network. Our discussion was about physical layer only. Now you suddenly come with OSI Level 2, 3 protocols, XML/JSON nonsense.  In other words: what you describe is way too overcomplicated for "DIY home automation".

author: Unixon
author_rank: Frequent Contributor
timestamp: November 15, 2020, 07:41:11 pm
content: Quote from: ogden on November 15, 2020, 07:31:32 pmNone of "super plug and play" and "super expandable" is about physical layer.

author: ogden
author_rank: Super Contributor
timestamp: November 15, 2020, 07:49:08 pm
content: Quote from: Unixon on November 15, 2020, 07:41:11 pmYes. I agree. - When we look at "plug and play" from computer engineer point of view. Plug and play in low end hardware not always means "autoconfiguration". Very often stage technicians say that DMX is plug and play because all you need is 1) set address using switches/jumpers 2) plug 3) playSame way in DIY home automation: KISS approach everywhere unless you want to build it forever and never finish.