packetizing data for serial to windows - Page 1
Forum section: electronics / general

author: Harvs
author_rank: Super Contributor
timestamp: May 15, 2013, 01:14:51 am
content: I'm working on part of project at the moment where I'm sending a serial stream to windows. In the past I've just used a basic packet scheme with non-unique start word and checksum.In this case I need to capture all the data without missing packets, data rate is going to be a max of about 1kB/s of raw data, so it does leave plenty of time for resend at 115200b/s.I was thinking of adding a sequential packet number, and just getting windows to ack every packet with a byte back to the uC.I'm wondering if anyone has done this before, is there a significant issue with windows having its occasional "moments" when it decides to auto-update and the like? I'd prefer if I could keep the windows side of it to a C# app in .NET with no custom drivers.Adding buffering schemes where the uC had to keep track of what packets had been received and what hadn't would significantly increase the code complexity and size on the uC.

author: NiHaoMike
author_rank: Super Contributor
timestamp: May 15, 2013, 01:23:13 am
content: Use CTS/RTS flow control.

author: ecat
author_rank: Frequent Contributor
timestamp: May 15, 2013, 01:54:46 am
content: I would turn your proposed solution around and have Windows initiate a request to which the uC responds, if windows fails to receive a reply within a certain time it initiates another request for the same packet, if the reply is received in a timely fashion Windows can request the next packet.Why do it this way?It's usually simple enough to ensure the uC can respond to a request in time 't', as you point out it is impossible to ensure Windows will respond within a specific time - unless that time is sufficiently long but even then what happens if the PC gets switched off or the cable is disconnected? All in, making Windows the comms master is generally the preferred solution.Packet synchronisation:A windows request contains a packet id byte or word.If the packet id == 0, the uC returns with the oldest packet and the id of the next packet to be read.No valid uC packet can have an id == 0.If the uC receives a request for a non-existent id it returns the oldest packet and the id of the next packet to be read.If the packet id received by Windows differs from the one requested Windows can flag an error and optionally resync its id with the one received.If Windows fails to receive a reply within time 't' it retries X times with the same packet id. If no reply is received after X retries Windows flags an error and optionally resets its id back to zero before recommencing the requests.I think that covers the eventualities. As to how and what and how many packets the uC buffers in the case of no requests, that's up to you.

author: Rufus
author_rank: Super Contributor
timestamp: May 15, 2013, 02:59:18 am
content: Quote from: Harvs on May 15, 2013, 01:14:51 amUnless you are going to store a load of packets while they are not acknowledged it would be simpler and as effective to just send the most recent 'n' packets repeatedly and let the PC ignore duplicates and notice gaps. You link rate is about 10 times your data rate so it would work for 'n' up to about 10.

author: Harvs
author_rank: Super Contributor
timestamp: May 16, 2013, 02:16:25 am
content: Thanks for the suggestions. I may have been a bit too quick to dam windows to hell.I implemented the basic scheme with an error detection word and sequential packet numbering, streaming adc conversions to a mschart object and monitoring the frames for errors. I had this running for several hours whilst doing various tasks on the pc including some cad rendering. Other than the first packet, it didn't miss one over the whole time. I was actually expecting it to drop a couple of packets per min.

author: flynnjs
author_rank: Contributor
timestamp: May 16, 2013, 07:53:19 am
content: Quote from: Harvs on May 16, 2013, 02:16:25 amApplications shouldn't affect the ability of an OS to service driver interrupts.Other heavy I/O could tho, so I would try loading up with those sort of tasksrather than CPU intensive tasks and see how you fair then.But, as someone else commented, HW flow control might be a betterindication to the uC to start buffering because the host is busy.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: May 16, 2013, 08:59:10 am
content: As real COM ports are getting rare, you need to consider the effects introduced by a USB-serial interface. An issue with acking every packet is latency introduced during the USB-Serial conversion process, which can slow things down a lot. USB interfaces tend to have timeouts to decide when to forward serial data to the host - configuring these correctly can make a huge difference to throughput, as can careful choice of packet size.there will typically be reasonable sized buffers in the USB driver, so you shouldn't need to do handshaking down at the byte level - you ideally want to be sending fairly large packets, but you still have the issue of timeouts & latency on the leftovers between the last full USB packet and the last byte of your payload.If the overall amount of data isn't that much you can probably get away with checking fairly infrequently for possible missed data, as overrun errors will be fairly unlikely.

author: Harvs
author_rank: Super Contributor
timestamp: May 16, 2013, 10:04:19 am
content: Quote from: mikeselectricstuff on May 16, 2013, 08:59:10 amI should have stated that from the start, a serial to USB converter is what I'm using.Sending larger packets works well from the transfer speed perspective (making use of DMA in the uC), but I still want to achieve around 15-20 frames/sec to provide a good update rate on the screen. However, the overall data rate is pretty low, so I haven't seen any problems yet.

author: Psi
author_rank: Super Contributor
timestamp: May 16, 2013, 11:06:08 am
content: Quote from: mikeselectricstuff on May 16, 2013, 08:59:10 amJust of of interest, what size would you guys define as a 'fairly large packet'?I have to rewrite some serial coms code soon and packet size is something i will be looking at.Currently the packets are 20 bytes long and its at 57600 baud

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: May 16, 2013, 11:51:54 am
content: Quote from: Psi on May 16, 2013, 11:06:08 amA few hundred bytes. You will find thet as packet size increases, you will see sudden jumps in latency/transfer speed as you get close to the USB frame size (don't recall offhand what this is ). Fiddling with latency timer settings can make a huge difference - the USB chip waits til either it has a full buffer, or the timer expires before sending the data to USB. The FTDI driver can also be programmed to force transmission on receipt of a special terminating byte value.I'm sure FTDI have some appnotes on optimising efficiency.

author: ToBeFrank
author_rank: Regular Contributor
timestamp: May 16, 2013, 02:29:14 pm
content: Quote from: Harvs on May 16, 2013, 10:04:19 amUse RTS/CTS as has been already stated. Also, if it's an FTDI device, use their D2XX driver rather than the VCP driver. Using the D2XX driver you can configure the timeouts and such for your packet size. I was able to achieve very close to the same frame rates as a real serial link without doing any buffering of packets. My packets are tiny, ~20 bytes.

author: firewalker
author_rank: Super Contributor
timestamp: May 17, 2013, 10:25:24 am
content: Quote from: mikeselectricstuff on May 16, 2013, 08:59:10 amUsing something likelibusbXwould be "better"/faster?Alexander.