Can boxcar averaging cause rounding error? - Page 1
Forum section: electronics / general

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 03:41:43 am
content: Got a setup where incoming pulses of variable frequency cause either a high or a low port pin output depending on whether the input frequency is above or below a certain frequency. Just sampling the counter, resetting it, then comparing it to a fixed value. No big deal. Without averaging it works dead nuts on. Perfect.But if I boxcar average the last sixteen 16 bit counter samples by adding them up then right shifting this total four times, the frequency trip threshold is very slightly low. Instead of tripping at 120Hz it trips at something like 119.5Hz IIRC. I should run it again and get an exact figure. Are the four LS bits that are lost in the right shift the cause of this? It's no big deal, just want to know what's going on.

author: Phoenix
author_rank: Frequent Contributor
timestamp: May 14, 2024, 03:59:20 am
content: A right shift is a division with floor rounding. In order to correctly round you need to add half the divider to the start value.Code:value = (value+8)>>4

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 04:18:00 am
content: Quote from: Phoenix on May 14, 2024, 03:59:20 amThat is super interesting! I didn't know that. Thanks!

author: newbrain
author_rank: Super Contributor
timestamp: May 14, 2024, 08:59:53 am
content: A couple of notes:1. I prefer to be explicit when coding (say what you mean, not what you think the compiler should be doing), so I'd use /16 rather than >>4 when I mean math and not bit manipulation. If the compiler is not a toy,it will use a shiftevenwithout optimizations(but see below). It's also easier to update, should the divisor change.2. One of the improvement of C99 over C89 is the clear definition of integer division: a / will always round towards 0 (so -15 / 4 == -3) while a shift with a negative first operand is implementation defined. Other languages might differ: (ANSI) FORTH, e.g., rounds towards -∞.3. Most implementations will perform an arithmetic shift on a signed operand, so the result will be rounded towards -∞ (so -15 >> 2 == -4), but the standard does not prevent using a logical shift (same as with an unsigned argument, so -15 >> 2 == 1'073'741'820, with surprising results).Points 2 and 3 above are a bit moot in this case, as the OP is averaging a frequency - I don't expect it to take a negative value in this context.

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 11:27:05 am
content: Time for an update.I should have mentioned, this thing is written in assembly, (NXP S08 micro) so the logical shift right isn't doing anything unexpected.I am looking for 120Hz so 8,33333 mS.1.25MHz counter so 8.33333 mS = 10,417 closest integer count for comparison.I am polling the input pin with a 500nS loop so there is a small amount of latency. As a result, I am judging the threshold point as when the output pin dithers with approximately a 50% duty cycle.No average = 119.99 HzAverage 16 = 119.085 HzAverage 16 with 8 added to sample = 119.175 HzSo it is better, but not much. Hmmm...

author: mtwieg
author_rank: Frequent Contributor
timestamp: May 14, 2024, 12:16:39 pm
content: Accumulating the 16 samples first and then dividing by 16 (with the +8 to ensure rounding) should also work, assuming it doesn't overflow your data type.If it's only 16 samples, should be straightforward to look at the raw samples (as integers, not Hz) and check things by hand.

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 01:11:50 pm
content: Quote from: mtwieg on May 14, 2024, 12:16:39 pmI've done exactly that. I've got 24 bits to contain the total, and each sample can't exceed #46875 so there's plenty of room. Here's the code. I've cut some irrelevant parts out. Don't want to give things away. Besides, I'm no hotshot coder.Code:********************************************************************************MAIN_LOOPLDHX#1250;3 1mS delay to preventNO_RETRIGAIX#-1;2 immediate retriggerCPHX#0;3 8 cyc 0.8uS loopBNENO_RETRIG;3WAIT_TRIGBRSETTRIG,PORTD,WAIT_TRIG;trigger on low pulseLDHXTCNTH;get counterSTATCNTH;reset counterAIX#8;add 8 to sampleSTHXTEMP_CTR_MSBJSRPUSH_DOWN;push old counter readings down one placeBRSETTOF,TSC,TOO_SLOW;if under 37.5mS and set flagBCLRSLOW,FLAGSLDHXTEMP_CTR_MSB;latest countCONTINUESTHXMSB_1;update to latestBCLRTOF,TSC;just in caseJSRADD_UP;add up last 16 counter periodsJSRDIV_16;average = MSB_TOT:LSB_TOTBRSETSLOW,FLAGS,NO_AVG; > 37.5mS so don't use average*********************************************LDHXMSB_TOT;averaged figureCPHX#10417;120Hz, 8.333mSBHSNO_BOOSTBSET4,PORTB;hi outputJMPMAIN_LOOP*********************************************TOO_SLOWLDHX#46875BSETSLOW,FLAGSJMPCONTINUE*********************************************NO_AVGMOVVOLTS_14,PORTC;So first slow pulse won't haveJMPMAIN_LOOP;to wait for 16 averages.*********************************************NO_BOOSTBCLR4,PORTB;low outputJMPMAIN_LOOP********************************************************************************;push old counter readings down one placePUSH_DOWNLDHXMSB_15STHXMSB_16LDHXMSB_14STHXMSB_15LDHXMSB_13STHXMSB_14LDHXMSB_12STHXMSB_13LDHXMSB_11STHXMSB_12LDHXMSB_10STHXMSB_11LDHXMSB_9STHXMSB_10LDHXMSB_8STHXMSB_9LDHXMSB_7STHXMSB_8LDHXMSB_6STHXMSB_7LDHXMSB_5STHXMSB_6LDHXMSB_4STHXMSB_5LDHXMSB_3STHXMSB_4LDHXMSB_2STHXMSB_3LDHXMSB_1STHXMSB_2RTS*************************************ADD_UPCLR LSB_TOTCLR MSB_TOTCLR EXT_TOTLDA LSB_TOTADD LSB_1STA LSB_TOTLDA MSB_TOTADC MSB_1STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_2STA LSB_TOTLDA MSB_TOTADC MSB_2STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_3STA LSB_TOTLDA MSB_TOTADC MSB_3STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_4STA LSB_TOTLDA MSB_TOTADC MSB_4STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_5STA LSB_TOTLDA MSB_TOTADC MSB_5STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_6STA LSB_TOTLDA MSB_TOTADC MSB_6STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_7STA LSB_TOTLDA MSB_TOTADC MSB_7STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_8STA LSB_TOTLDA MSB_TOTADC MSB_8STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_9STA LSB_TOTLDA MSB_TOTADC MSB_9STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_10STA LSB_TOTLDA MSB_TOTADC MSB_10STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_11STA LSB_TOTLDA MSB_TOTADC MSB_11STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_12STA LSB_TOTLDA MSB_TOTADC MSB_12STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_13STA LSB_TOTLDA MSB_TOTADC MSB_13STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_14STA LSB_TOTLDA MSB_TOTADC MSB_14STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_15STA LSB_TOTLDA MSB_TOTADC MSB_15STA MSB_TOTJSREXT_ADDLDA LSB_TOTADD LSB_16STA LSB_TOTLDA MSB_TOTADC MSB_16STA MSB_TOTJSREXT_ADDRTS*************************************EXT_ADDLDA EXT_TOTADC #0STA EXT_TOTRTS*************************************DIV_16MOV#4,LOOP_CTRDIV_LOOPLSR EXT_TOTROR MSB_TOTROR LSB_TOTDBNZLOOP_CTR,DIV_LOOPRTS****************************************************************************************************************************************************************ORG $FFFEFDB START********************************************************************************

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 14, 2024, 07:14:14 pm
content: Shouldn't yousetLSB_1 at some point, too?

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 11:15:45 pm
content: Quote from: Nominal Animal on May 14, 2024, 07:14:14 pmIt is inherent. The upper and lower bytes of H:X are stored at MSB_x:LSB_x. Provided that the latter are at adjacent memory locations, which they are.

author: Circlotron
author_rank: Super Contributor
timestamp: May 14, 2024, 11:22:09 pm
content: Quote from: Nominal Animal on May 14, 2024, 07:14:14 pmIt gets set at the CONTINUE label either to the latest sampled counter value or to a fixed maximum value of #46875. The latter is a consequence of the timer overflow flag being set so it goes through the TOO_SLOW label.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 15, 2024, 12:12:14 am
content: Quote from: Circlotron on May 14, 2024, 11:15:45 pmWhat architecture and instruction set is this?Wouldn't it make more sense to use a 24-bit accumulator you add the result to, instead of a 16 byte buffer? Or do you really need the update rate of a sliding window?

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 01:25:37 am
content: NXP S08 architecture. Specifically 9S08FL micro.Doing it in assembler so what is a different way to have a 24 bit accumulator other than what I have done?The averaged value I am getting seems to be about 80 counts edit -> *less* than expected.

author: KE5FX
author_rank: Super Contributor
timestamp: May 15, 2024, 01:59:20 am
content: Normally rounding error just shows up as DC offset, not frequency error. Could there be an aliasing effect between the clock frequency and the frequency being measured? To rule that out, try some other frequencies and see how they influence the magnitude and sign of the error.Hysteresis is another thing to consider. A ZCD typically shouldn't actually be implemented as a "zero crossing detector," since that's where the noise is. Instead look for falling and rising edges separately with nonzero threshold voltages, if you're not doing so already (I didn't read the code.)

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 02:10:40 am
content: No ZCD involved. Nothing to do with power line frequency either. I’m looking at the repetition rate of a 20uS duration pulse.

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 05:11:53 am
content: Hey, I got it!In the RAM equates list the MS byte for each 16 bit sample is listed first because whenever the H:X register is loaded from or saved to memory you just specify the MS byte location and the LS byte is the next one along. e.g. LDHX MSB_15 and it grabs MSB_15 and LSB_15.Where I messed up was the 3 byte totals RAM. I had it listed as LS, MS, EXT(ra) so after the samples were all added up and LSR'd four places to the right, H:X instead of being loaded with MS_TOT:LS_TOT was being loaded with MS_TOT:EXT_TOT, and seeing these three had been LSR'd that meant EXT_TOT (the wrongful LS byte being loaded into X) was empty so that would account for the approximately 80 lower count than expected.Code:MSB_15EQU$5CLSB_15EQU$5DMSB_16EQU$5ELSB_16EQU$5FLSB_TOTEQU$60MSB_TOTEQU$61EXT_TOTEQU$62FLAGSEQU$63Something else, it is actually more accurate without adding 8 to each sample as discussed earlier.

author: mikerj
author_rank: Super Contributor
timestamp: May 15, 2024, 06:48:45 am
content: Quote from: Circlotron on May 15, 2024, 05:11:53 amI'm not surprised, you aren't supposed to be adding 8 toeach sample! You add 8 to the total of 16 samples before dividing by 16. This gives you rounding to the nearest integer rather than just truncating all fractional bits.

author: jpanhalt
author_rank: Super Contributor
timestamp: May 15, 2024, 12:01:25 pm
content: @CirclotronCan you post the 16 values used in post #5?John

author: S. Petrukhin
author_rank: Super Contributor
timestamp: May 15, 2024, 12:11:07 pm
content: Quote from: Phoenix on May 14, 2024, 03:59:20 amIs this true for negative values?

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 12:21:07 pm
content: Quote from: jpanhalt on May 15, 2024, 12:01:25 pmThey would be in the region of 10,417 if the measure pulse rate was 120 Hz. Lower frequency = longer time between pulses so greater count value. Counter rate = 1.25MHz.

author: gf
author_rank: Super Contributor
timestamp: May 15, 2024, 12:50:49 pm
content: Quote from: S. Petrukhin on May 15, 2024, 12:11:07 pmIt fails for very large positive values. For eample, if type is int, then values >= MAX_INT - 7 lead to a negative result.

author: jpanhalt
author_rank: Super Contributor
timestamp: May 15, 2024, 12:56:26 pm
content: I was referring to this:Quote from: CirclotronAn alternative way to round, which I am sure you saw, is to check the carry bit after the last rotation (e.g., PIC equivalent is decrease counter and skip if zero) and add 1, if set. My experience is limited to PIC Assembly, so the instructions and execution times are a bit different.John

author: S. Petrukhin
author_rank: Super Contributor
timestamp: May 15, 2024, 01:12:34 pm
content: Most often I use the moving average method according to the formulaOvrValue:=(1-k)*OvrValue+k*CurValuewhere k is usually 0.2 or 0.1.This smooths out fluctuations quite well, recovers quickly and does not require accumulation.If using real mathematics is inconvenient, you can make a shift towards integer values.

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 01:17:59 pm
content: Quote from: jpanhalt on May 15, 2024, 12:56:26 pmIf I understand you, the 16 values would be 1.25 MHz x the period of each of those frequencies.

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 01:20:32 pm
content: Quote from: jpanhalt on May 15, 2024, 12:56:26 pmHey, That's interesting! Thanks for that. It took a moment to sink in (11:20pm here) but yeah, that's cool!

author: S. Petrukhin
author_rank: Super Contributor
timestamp: May 15, 2024, 01:23:01 pm
content: Quote from: CirclotronJust add up the current values as they are, and then round up the result.The sum of rounded values will never equal the rounding of the average value.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 15, 2024, 03:48:16 pm
content: Quote from: S. Petrukhin on May 15, 2024, 12:11:07 pmGood point.It rounds halfway upwards, which is towards zero for negative values.One needs to subtract half the divisor for negative values for rounding halfway away from zero; i.e.value = (value < 0) ? ((value - 8) >> 4) : ((value + 8) >> 4);Note that this applies to all division operations, not just arithmetic shift right:value = (n < 0) ? (n - d/2) / d : (n + d/2) / d;This even works for negative divisorsd, noting that the addend (±d/2, which I typically callhalf) has the same sign as the divisor ifnis positive, negated ifnis negative.For integer calculations, this works best whenhalfitself is truncated towards zero (no rounding applied), so that looking at the results of all possible numerators, you'll have|d|zero results centered atn=0(|d|-1if|d|is even), and then|d|consecutive same results in both directions. For example, ford=3,(n < 0) ? (n-1)/3 : (n+1)/3yields results -3, -2, -2, -2, -1, -1, -1, 0, 0, 0, 1, 1, 1, 2, 2, 2, 3 forn=-8..+8. Ford=4,(n<0) ? (n-2)/4 : (n+2)/4yields -2, -1, -1, -1, -1, 0, 0, 0, 1, 1, 1, 1, 2 forn=-6..+6.For a sequence of values, you add thehalfto thesum, not to each term:sum = t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9 + t10 + t11 + t12 + t13 + t14 + t15 + t16;avg = (sum < 0) ? (sum - 8)/16 : (sum + 8)/16;Mathematically,$$\left\lceil \frac{1}{N} \sum_{i=1}^N t_i \right\rfloor = \left\lfloor \frac{1}{N} \left( H + \sum_{i=1}^N t_i \right) \right \rfloor, \quad H = \pm \left\lfloor \frac{N}{2} \right\rfloor$$where \$H\$ is negated if and only if the sum is negative.If you do the division and roundingfirst, you lose precision, and introduce excess rounding error. It is easiest to understand if you consider decimal numbers rounded to nearest integer. \$\lceil 0.3 + 0.3 + 0.3 + 0.3 + 0.3 + 0.3 \rfloor = 2\$, but \$\lceil 0.3 \rfloor + \lceil 0.3 \rfloor + \lceil 0.3 \rfloor + \lceil 0.3 \rfloor + \lceil 0.3 \rfloor + \lceil 0.3 \rfloor = 0\$.

author: Andy Watson
author_rank: Super Contributor
timestamp: May 15, 2024, 04:11:47 pm
content: Unless there is a need for the averaged value I would skip the "dividing" and perform the limit checks against \$16 \times n\$

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 15, 2024, 04:20:16 pm
content: Quote from: Andy Watson on May 15, 2024, 04:11:47 pmThat way no rounding is needed either.The only downside is that you need to compare against the sum, which is 4 bits larger than the individual terms.If each term fits in 0..15 or -8..7, then 8-bit sum and limit values suffice.If each term fits in 0..4095 or -2048..2047, then 16-bit sum and limit values suffice.If each term fits in 0..1048575 or -524288..524287, then 24-bit sum and limit values suffice.If each term fits in 0..268435455 or -134217728..134217727, then 32-bit sum and limit values suffice, and so on.

author: Circlotron
author_rank: Super Contributor
timestamp: May 15, 2024, 11:58:11 pm
content: Quote from: Andy Watson on May 15, 2024, 04:11:47 pmThat would work in a way, but seeing the limit check is +/- 1 in both cases, the averaged /16 value gives me some low pass filtering, so any frequency jitter in the sampled pulses won't cause as much dithering about the threshold. I would effectively have a 16 count dead zone that drifts up or down.I suppose I could achieve much the same outcome by using the undivided total and a bit of hysteresis on the limit, but the idea of averaging appealed to me. Some might say that subjectivity has no place in software, others might disagree. If two different approaches give the same end result with no downside either way, then I choose the one that gives the most satisfaction.

author: Circlotron
author_rank: Super Contributor
timestamp: May 16, 2024, 01:19:43 am
content: Quote from: Phoenix on May 14, 2024, 03:59:20 amHaving thought about this overnight, the purpose of adding half the divisor becomes clear. It seems comparable to how an A/D converter assigns an analog value of say 3 to the digital span of 2.5 to 3.5 so it is centered on 3. It is offset by 1/2 LSB, same as the added 8 count is divided down to effectively 0.5 count in my case.

author: jpanhalt
author_rank: Super Contributor
timestamp: May 16, 2024, 08:24:47 am
content: In Assembly you should have a register that does what STATUS in PIC does. That is, flags for carry, zero, and maybe other things, like DC (carry between nibbles).Adding one half of the divisor to the sum before dividing is one way to ensure rounding. Look at the following example:1) In decimal, add 3.0+3.5+4.0+4.5 = 15. The average is 3.75, so you round to 4 to avoid the decimal.2) d.15 +2 = 17. Divide by 4 = 4.25 --> 4 rounded .3) d.15 = b'1111'. Rotate right once --> b'0111' w/ carry =1. Rotate again to divide by 4 -->b'0011' w/ carry =1. Now add carry to the result = b'0100' = 4 . Same answer. Why? The bit in carry after 2 rotations is the 2^1 bit (i.e., 1/2 the divisor).Coding in Assembly is pretty easy either way. I don't do C, but have been told using the STATUS register is not as clear cut.

author: S. Petrukhin
author_rank: Super Contributor
timestamp: May 16, 2024, 08:34:50 am
content: Quote from: Nominal Animal on May 15, 2024, 03:48:16 pmIt seems to me that you have forgotten the format of negative numbers.The operator ">>" (shr) for a negative number will shift the sign and the number will cease to be negative.

author: Nominal Animal
author_rank: Super Contributor
timestamp: May 16, 2024, 09:01:23 am
content: Quote from: S. Petrukhin on May 16, 2024, 08:34:50 amNo; it is implementation defined in the C standard (C99-C23 6.5.7p5).GCC, SDCC, Clang, ICC, and all other C compilers I've used implement(v >> n)as an arithmetic right shift for signed integer types ofv, shifting in copies of the sign bit, iff the target architecture has an arithmetic right shift operation. It does mean that with these compilers (and in e.g. Bash),(-1)>>1 == -1, which can be confusing.I haven't seen any C compilers that shifted zero bits in when using>>for negative signed integer types, but the ones I've checked all do have separate arithmetic and binary right shift instructions (and exact-width binary integer types with signed integers using two's complement format).

author: S. Petrukhin
author_rank: Super Contributor
timestamp: May 16, 2024, 09:14:22 am
content: Quote from: Nominal Animal on May 16, 2024, 09:01:23 amThank you!I didn't know the subtleties of С.There is no >> in Pascal (where I live), there is an explicit shr, which is an ordinary bit shift.

author: SiliconWizard
author_rank: Super Contributor
timestamp: May 16, 2024, 10:27:38 pm
content: Bitwise manipulation of signed integers -unless you have a good rationale to do it, and know/can rely on well-defined behaviors of bitwise operators on them in the language you use- is IMHO a bad idea in most cases anyway.

author: jpanhalt
author_rank: Super Contributor
timestamp: May 16, 2024, 11:10:42 pm
content: Quote from: SiliconWizard on May 16, 2024, 10:27:38 pmProbably wise, but my routine in PIC Assembly for division (rotate and subtract, modified from a PICList link) involves negative numbers to avoid having to undo a subtraction. I do not use the routine when the dividend or divisor are negative. I haven't tested it in that situation but suspect it would fail. "Most" is key. In this thread, the TS has said the numbers are all positive.

author: mikerj
author_rank: Super Contributor
timestamp: May 17, 2024, 09:10:44 am
content: Quote from: jpanhalt on May 16, 2024, 11:10:42 pmTo get an arithmetic right shift on the PIC, first rotate the MSB left one bit but don't write the result back to the variable, then perform the right shifts on your value:Code:RLF myval_msb,W ; put MS bit into carryRRF myval_msb,F ; Shift MSB right, sign bit is shifted from carry into bit 7RRF myval_lsb,F ; Shift LSB rightIf you clear carry first you get a logical shift, which will not work as a division on negative values.

author: jpanhalt
author_rank: Super Contributor
timestamp: May 17, 2024, 09:40:50 am
content: Microchip has added arithmetic shifts to its 8-bit instruction set. See:http://datasheet.elcodis.com/pdf/16/18/161877/pic16f1936-iss.pdf(Apologies for not being a direct link. MCC seems to want to hide it behind marketing and sign-in requirements)The code I mentioned for division is for positive integers and can me easily modified for any number of bytes. I have not tried it beyond 32-bit by 32-bit division. Usually, I us it for 32-bit by 16-bit or fewer division.