[solved] Linux/Arm, catastrophic crashes - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: October 01, 2021, 10:41:31 pm
content: I have tested several kernel from 5.4.128 to 5.15.0, and the result is always the same:- within 48h of burn-in testing- the kernel crashesCode:neo / # XFS (sda3): Metadata corruption detected at 0xc037735c, xfs_inode block 0x953140 xfs_inode_buf_verifyXFS (sda3): Unmount and run xfs_repairXFS (sda3): First 128 bytes of corrupted metadata buffer:00000000: 6d 20 25 54 20 25 73 20 25 54 0a 00 46 6f 75 6e m %T %s %T..Foun00000010: 64 20 6e 65 77 20 72 61 6e 67 65 20 66 6f 72 20 d new range for00000020: 00 00 00 00 41 73 73 65 72 74 69 6f 6e 73 20 74 ....Assertions t00000030: 6f 20 62 65 20 69 6e 73 65 72 74 65 64 20 66 6f o be inserted fo00000040: 72 20 00 00 0a 09 42 42 20 23 25 64 00 00 00 00 r ....BB #%d....00000050: 0a 09 45 44 47 45 20 25 64 2d 3e 25 64 00 00 00 ..EDGE %d->%d...00000060: 0a 09 50 52 45 44 49 43 41 54 45 3a 20 00 00 00 ..PREDICATE: ...00000070: 0a 41 53 53 45 52 54 5f 45 58 50 52 73 20 74 6f .ASSERT_EXPRs toXFS (sda3): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x953140 len 32 error 117XFS (sda3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.XFS (sda3): xfs_do_force_shutdown(0x8) called from line 3749 of file fs/xfs/xfs_inode.c. Return address = 7769d00aXFS (sda3): Corruption of in-memory data detected. Shutting down filesystemXFS (sda3): Please unmount the filesystem and rectify the problem(s)Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000007I have exhaustively tested- the ram: all tests passed- the hard-drive (and tested four different models): all tests passed with badblocks and SMART- the operating temperature is 22C thanks to a big heat-sink + fan coolingSo I'm running out of ideasthe burn-in test consists of- 8GB of swap activity, moving up and down the virtual memory over 512MB of physical ram- 90% of CPU activity, but only 2 of 4 cores used- the FPU is not usedThe system appears to work "ok" for 12h, but within 48h it crashes. The problem here is what and how to investigate.(edit: I seriouslydon't recommendAllwinner's stuff)

author: DiTBho
author_rank: Super Contributor
timestamp: October 01, 2021, 10:45:38 pm
content: it happens (tested) with- ext4- ext2- xfs-v4- btrfsSo I can rule out any filesystem bug, rather it smells something bad with swap? or with the EHCI-bulk module. The storage is a USB to sATA hard-drive connected to the first EHCI controller.

author: DiTBho
author_rank: Super Contributor
timestamp: October 01, 2021, 10:48:28 pm
content: It is catastrophic because it severely damages the file system to the point that the partition must be wiped, formatted and restored from backup.

author: Marco
author_rank: Super Contributor
timestamp: October 01, 2021, 11:35:13 pm
content: Could the power be dipping?

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 08:43:22 am
content: Quote from: Marco on October 01, 2021, 11:35:13 pmThe PSU I am using is a professional laboratory power supply manufactured by Rigol.The SoM module eats << 400mA @ 5V, the max peak is 600mA, the current is limited to 1A.The hard-drive is powered by the second channel of my PSU, it eats 1.1A @ 5V as max peak (never measured, but it's what Seagate said, I see less current absorbed, anyway) the current is limited to 2A.What makes me perplexed: I have enabled all the kernel "debugging" and "verbose" support (even the memory leak checkers and SCSI verbose logs) and can successfully run exhaustively read/write tests that continuously moves 50Mbyte/sec for 96hours without a single clue of crash.As well as I can mount 400Mbyte as ramdisk and successfully run exhaustively read/write tests that continuously moves 300Mbyte/sec for 96hours without a single clue of crash.

author: Twoflower
author_rank: Frequent Contributor
timestamp: October 02, 2021, 08:50:43 am
content: If its Hard-Disk as real spinning disk, I would expect higher peak currents in case the heads move around. They're probably very short. Attach a scope to the 5V HDD supply, in single-shot with trigger level around 4.5V (I would expect 5V -10% is still accepted) and wait to see if voltage drips happen.

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 11:05:06 am
content: Quote from: evb149 on October 02, 2021, 09:25:02 amThat's a good idea.

author: PKTKS
author_rank: Super Contributor
timestamp: October 02, 2021, 11:26:46 am
content: There has been issues with swap in kernel 5x branch.Some were claimed fixed although depending on the hardware itself I would not put that into production status.For the sake of eliminating any chance of branch you can (if possible by hardware) check the ultra stable production quality 4.19.x branch or other close.If your hardware does not require anything that new pretty much your chances will improvePaul

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 11:32:41 am
content: Quote from: Twoflower on October 02, 2021, 08:50:43 amYes, it's a 2.5" ElectroMechanical Hard Disk Drive spinning at 5400 rpm.When one month ago I exhaustively tested the usb-bulk, I used progressive LBA addresses to read write read-back blocks (this is also what the program "badblocks" does), but if the problem is related to high current peak when heads move around, that means I need to run tests with random LBA addressing, which is the case of what the kernel does with the virtual memory, otherwise progressive LBA addresses correspond to progressive slow head movements, therefore less current peak.Mumblebut it somehow makes sense, I mean the virtual memory I observed during the kernel stress-test moved the heads like a deranged dog on speed (I heard the sound the heads made, kind of "tic, tic, tic, tac, tac, tic"), and probably at some point this may require an high peak of current that the on-board LDO is unable to provide.Umm, so a problem with the power on-board module, probably a tanker-capacitor or something similarI will investigate and also try with a Solid state disk (flash-SSD or sdram-disk). But first I want to re-test everything on a virtual machine to make sure it's not a kernel bug.(oh, here I assume the rootfs cannot crash the kernel, but it's a strong conjecture)Thanks guys! Great advice!

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 11:40:54 am
content: Quote from: PKTKS on October 02, 2021, 11:26:46 amThat's worrying but it also explains why individual tests pass, but the system crashes during swap activities.So it also makes sense to re-try everything with k4.19.*Thanks!edit:I also tested K5.14.1, no progress for me.

author: ejeffrey
author_rank: Super Contributor
timestamp: October 02, 2021, 06:01:26 pm
content: Quote from: PKTKS on October 02, 2021, 11:26:46 amDo you have a citation for that? Is that platform specific?

author: ejeffrey
author_rank: Super Contributor
timestamp: October 02, 2021, 06:24:16 pm
content: You say you have exhaustively checked the ram. How? For how long? Have you tried walking back the memory timings?How many of these units do you have? Does it occur on multiple systems or just one?What is the die temperature? Surely not 22 C, maybe the chip has crappy thermal packaging and isn't cooling enough or thermal throttling properly? Try backing down the clock speed?

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 02, 2021, 07:07:41 pm
content: Quote from: DiTBho on October 02, 2021, 08:43:22 amIn my experience, SBC current use is very spiky, and they don't typically have much bulk capacitance on the power rail.(This is basically the reason I started developingthisTeensy 3.2 carrier for Odroid HC1. It takes a strictly regulated 5V in, but when using a 3.5" spinny-rust HDD, can consume up to 4-6A. The idea was to actually carefully measure whether there were any sub-microsecond dips in the voltage due to current use spiking. Such minimally short dips sound like they shouldn't affect anything, but remember, these SBCs really don't have any bulk capacitance on their power inputs to smooth them out, not at these current use levels. And on a lab power supply, there usually isn't much bulk capacitance on the output either, so you're more likely to see this on a lab power supply than on a typical 5.15V wall warts that tend to have at least some bulk capacitance as an output filter.)(The scenario I'm thinking is that various actions cause small current use spikes, and it is only a rare occurrence when the SBC operations combine just right to cause a deep enough dip: a very short current draw spike, that drops the power rail just enough to cause a kernel oops, perhaps due to a memory glitch, or a support chip glitch. I don't think the SoC chip is too sensitive to these, especially if they use switchers (SMPS) for their various power rails. The symptoms do fit, but this is less likely than a pure software (kernel) bug; it is just one thing to rule out as a possible cause.)I suggest ruling this out, by putting some bulk capacitance between your SBC input rails, say a couple of milliFarads of low-ESR capacitance (electrolytics) between the input power rail and ground (like bypass caps on chips), and re-test for a few days to see if this affects the stability of your SBC. It takes time, but is not difficult to do; and you can spend the test time poring over the kernel sources and LKML to see if anything similar (to the commonalities you've seen in the OOPSes when using different filesystems) has been reported. In other words, drudge work ensues...

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 08:05:04 pm
content: Quote from: ejeffrey on October 02, 2021, 06:24:16 pmYupQuote from: ejeffrey on October 02, 2021, 06:24:16 pmDifferent methods.u-boot memtestLinux "memtest/ramspeed" (it's a dedicated program)Code:Stuck Address : okRandom Value : okCompare XOR : okCompare SUB : okCompare MUL : okCompare DIV : okCompare OR : okCompare AND : okSequential Increment: okSolid Bits : okBlock Sequential : okCheckerboard : okBit Spread : okBit Flip : okWalking Ones : okWalking Zeroes : ok8-bit Writes : ok16-bit Writes : okQuote from: ejeffrey on October 02, 2021, 06:24:16 pm96hQuote from: ejeffrey on October 02, 2021, 06:24:16 pmI patched u-Boot with the lowest DRAM_CLK, and retested everything for 96h.All tests passed, and thanks to the new lower clock the ram-chip has a better temperature.Quote from: ejeffrey on October 02, 2021, 06:24:16 pmFour units with the same features (ram, disk, etc), tested in parallel. The same result.Each burn-in test is scheduled for two days; I launch the test in parallel in a corner of the lab, I wait for two days, then I check the results, which includes information on uptime and debugging (sadly scarce and yet any I / O profiling information).Quote from: ejeffrey on October 02, 2021, 06:24:16 pmThe thermal sensor inside the SoC reports 37 C (1), the thermal sensor near the heat-sink reports 22 C.The air temperature of the room is 19 C. I don't have an IR camera. The critical SoC temperature should be above 50 C.(1) highest reported temperature, with two cores at 90%, two core at 0-5%Code:source="/sys/class/thermal/thermal_zone0/temp"if [ -f $source ]thenecho $((`cat $source` / 1000))elseecho "$source not found"fiI disabled the speed stepping, the SoC always and only operates with the lowest CPU clock possible, which is 600Mhz instead of 1Ghz, and the GPU is disabled.

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 08:31:26 pm
content: Quote from: Nominal Animal on October 02, 2021, 07:07:41 pmConsidering that CF-Microdrive are more gentile, 2.5" HDDs spinning at 5400 rpm shouldn't have the same peak current you see in 3.5" HDDs spinning at 7200 rpm.My conjectureBut for instance my SCSI-SCA U320 Fujitsu Raptor spinning at 12000 rpm is literally a monster that makes a lot of noise when it starts because it sounds like a jet-turbine, and it's known to have high peak current (>> 5A) when the BLDC motor (4 phases) starts to spin and accelerate, in fact there is a "soft start" mode to manage it.I don't know, like you said, this stuff needs more measurements(thanks for advice, it's super useful)edit:seehere, High-speed drives by Seagate Barracuda take 8W on average with high current on-start-up / spin-up, but from the same brand Seagate a 2.5" mechanical drive consumes a bit less because the discs inside are smaller and lighter, so the motor doesn't need as much oomph, torque, to keep the discs spinning, and the spinning speed itself is 5400 vs 7200 rpm.

author: Twoflower
author_rank: Frequent Contributor
timestamp: October 02, 2021, 09:47:57 pm
content: I would be more worried to the head actuator. At the acceleration rates of the head-stack in modern drives they're reasonable power hungry. And thus will produce very spiky currents. Especially during frequent back and forth movements. Like copy on same drive, access lots of small areas spread over the whole platter(s).Edit:This paper shows the power consumption during seek (An Analysis of Hard Drive Energy Consumption, Picture 11). But the 2.5" you're using wont be that bad as the picture shows an older 3.5" drive.

author: DiTBho
author_rank: Super Contributor
timestamp: October 02, 2021, 10:11:11 pm
content: Quote from: Twoflower on October 02, 2021, 09:47:57 pmI am using 160GB and 320GB 2.5" sATA HDDs (~2010). Are modern enough? Or is it what you have with current 1TB ans 2TB HDDs?Quote from: Twoflower on October 02, 2021, 09:47:57 pmDamn, this exactly what the kernel does with virtual memory. A lot of fast copies with random LBA addresses.I am wondering if it is better to force a deferred I/O in the kernel to slow down the read/write operations on the swap and make them with more progressive LBA addresses; this means you have to add requests to a queue, sort it and process all the virtual memory requests later.I am also tempted to move the swap to the network, even if the NIC can only negotiate a 100Mbit/s link, which means 8Mbyte/sec, ~12 times slower than swap on a HDD partition.At the moment I am preparing the testing environment for k4.14.199, and I am compiling qemu-arm to test the kernel itself in a virtual machine.I will run this stuff on Monday.

author: DiTBho
author_rank: Super Contributor
timestamp: October 05, 2021, 10:32:23 pm
content: two machines with k5.15.1 + patches + hack have been running for 29 hours without a crash(fingers crossed)p.s.In "kernel next", I see disturbing patches for ext4 an kvm

author: DiTBho
author_rank: Super Contributor
timestamp: October 05, 2021, 10:35:45 pm
content: Two machines have been running k4.14.199 for 25 hours with a crashAnd I have a k4.14.199 running Qemu/armWe will see ...

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 05, 2021, 10:43:27 pm
content: Have you added a few milliFarads of bulk capacitance to the power input? I'd be very interested to know if it helps or not.

author: DiTBho
author_rank: Super Contributor
timestamp: October 06, 2021, 09:09:21 am
content: Quote from: Nominal Animal on October 05, 2021, 10:43:27 pmNot yetThis is the same hardware in the same configuration, just with different kernels.Yesterday I ordered qty=2 new boards and I will test them with bulk capacitance.Anyway, the sATA hard-drive has never been powered by the SBC, it has a dedicated power supply line since I am using my professional laboratory PSU to supply each board and each hard-drive independently.(I asked my boss to take five PSUs at home from work, and she agreed to lend me them. Nice girl! But in the future I will have to by a second Rigol PSU, for sure)Code:Rigol PSU___________| || PSU || || ch0 | ch1 || | || 12V | 05V || + - | - + ||_|_|_|_|_|_|| | | || +---+ || GND |Vcc12 Vcc05___________| | || HDD2.5"| |==== usb2| sATA | |=============== { Vcc05 , GND }|________|__|___________| || |======== usb2 (to sATA disk)| |======== eth0 (to testing machine)| SBC |======== uart0 (to testing machine)| ARM || || || |=============== { Vcc12 , GND }|___________|This means, the capacitor will help only if the problem is related to the SoC.With the k5.15.1 the temperature looks slightly higherCode:k5.15.1 # mytemperature34Code:k4.15.199 # mytemperature29I removed the cooling fan as it was powered by the SBC and I want to make sure there is no interference from any BLDC motor, the SoC now only works with the aluminum heat-sink.Code:# mycpufreq648Mhz@cpu0648Mhz@cpu1648Mhz@cpu2648Mhz@cpu3Other valid values for the PLL are 816Mhz and 1008MhzAll profiles are "power-save", and the speed-stepping is disabled, so this is a fixed frequency that never changes, and it's the same core clock frequency used in k4.15.199, but k5.15.1 probably different algorithms running hence they move more data therefore more transistors are switched per time therefore consume more current, therefore has slightly higher temperature.It makes sense

author: DiTBho
author_rank: Super Contributor
timestamp: October 06, 2021, 09:17:26 am
content: The latest stable k4.9 is k4.9.285.It has just been released.The updated 4.9.y git tree can be found at:Code:git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-4.9.yI will!

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 06, 2021, 09:57:21 pm
content: Quote from: DiTBho on October 06, 2021, 09:09:21 amRight.I guess using an oscilloscope in trigger mode (on the SBC power rail) could also be interesting. It doesn't have to have much bandwidth, since any glitches under say 1µs (corresponding to 1 MHz) should be filtered out. My own worries are glitches in the 1µs to 100ms range: long enough to not be fully filtered out due to lack of bulk capacitors on an SBC, but short enough to not cause a reliably triggered crash.A microcontroller like Teensy monitoring the voltage at say 100kHz rate (via a voltage divider and a buffer, like a TLV2371 RRIO opamp inmy design), can do this continuously, and collect statistical information –– that design for Odroid HC1 is such that the Teensy can be connected to a remote computer, and log both serial console and any voltage glitches: basically be an UART-to-USB adapter with two TTY endpoints on the host. The high-side current switch is there to act as a power-off switch, since the Odroid HC1 does not have one itself.I guess I should make a Teensy LC or Teensy 4.0 version carrier of that (perhaps leaving the current measurement and power switch sections optional?), since the 74LVC1T45 transceivers mean the SBC side UART voltage levels can be anything from 1.8V to 5.5V, as long as you have a suitable reference voltage. For this kind of testing (recording voltage glitches during a many-hour run), and interfacing to new/unknown devices like routers (due to the voltage shifters), it could be useful... The key being that on the host, you have one TTY endpoint directly to the SBC UART, and one that reports the voltage (continuously, or only changes or glitches) on the measured power rail (and can act as a control channel to the Teensy itself). (Using 16-bit binary, 100 kHz sample rate only takes about one fifth of the available practical bandwidth. All Teensies easily reach about 1,000,000 bytes/sec in Linux via TTY aka USB Serial even when using a cheap, USD $10 USB isolator; I've tested this in practice.)

author: DiTBho
author_rank: Super Contributor
timestamp: October 07, 2021, 04:10:55 am
content: The "uptime" is now > 48h, boards are all still fineProbably removing the cooling fan was a good idea, or completely unrelated to a deterministic cause, but during previous tests no single k5.1* kernel passed 24h burn-in when the cooling fan was in use: it was its tiny BLDC motor the root of this weird problem ... who knows?New patches also help. My goal is to get a working trust-able SBC, I am seriously late with this project. I wasted two months with Allwinner's bugs including bugs with their u-boot.It would be funny if the problem was the bldc motor of the cooling fan, which is attached on the SBC and sinks current from the same LDO that powers the SoC, but in this case you wouldn't see anything observing the SBC-power-rail-input of the board because the problem is internal the board and you should put the current and level monitor just after the LDO.

author: DiTBho
author_rank: Super Contributor
timestamp: October 09, 2021, 11:11:04 am
content: The uptime is now 4 days: tests all passedI think there is something really bad with the small fan cooler.

author: DiTBho
author_rank: Super Contributor
timestamp: October 11, 2021, 09:38:17 pm
content: so, in the meanwhile two new boards has arrived with the fan cooler installer so I tested everything in parallel with the same configuration/hard-drive brand/model, kernel version, and rootfs stuffresults:two new boards with fan cooler: both crashedtwo old boards without the fan cooler: still running (the uptime is now 6 days)I think it's the damn fan cooler, its BLDC motor probably sinks current with spikes and the LDO is unable to filter.

author: DiTBho
author_rank: Super Contributor
timestamp: October 11, 2021, 09:39:24 pm
content: Tomorrow I will instrument the LDO to measure things with a DSO

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 12, 2021, 06:39:47 am
content: Depending on the BLDC controller, it could even produce voltagespikes. Sharing the SoC supply with it is sheer idiocy! Even I – a bumbleduck hobbyist – would know enough to have a good filter separating the motor (any motor!) from the SoC supply.Depending on the supply voltage you use, perhaps you could use one of those small buck/boost converters to power a suitable PC case fan (3-pin voltage controlled one, 8 - 12 VDC) separately? I prefer larger ones over smaller ones, and undervolt them at 9-10v (depending on the fan), for quiet operation but sufficient airflow. You might consider 3D-printing a baffle and holder for the fan, to direct the airflow.

author: DiTBho
author_rank: Super Contributor
timestamp: October 12, 2021, 08:15:28 am
content: Quote from: Nominal Animal on October 12, 2021, 06:39:47 amYup. I haven't yet touch the hardware, but this for sure will be the first thing to hack and modify.Quote from: Nominal Animal on October 12, 2021, 06:39:47 amFrankly, I'm still shocked by this discovery. I mean, I don't have the board schematic, and the board itself looks so small that you need a magnifying glass to check things out.I thought like a user. if I buy a product, it should be well designed ... ain't it? umm.. No, wrong.In this case, it "emerged" that there is a problem, and the problem seems to be related to the cooling fan, and if this is pure idiocy ... well it's just the *last* idiocy ever with this board. Remember the on-sale choice to overclock the dram and - worse still, in a 1.2Ghz SoC sold without a head-sink? All default incorrect choices that you (the customer, the end user) need to correct.Quote from: Nominal Animal on October 12, 2021, 06:39:47 amWith my patches both u-boot and the kernel set the operating cpu-clock and the dram-clock to the minim, and in this configuration I have just demonstrated that the SoM doesn't overheat with a passive heat-sink, so in theory, you don't even need a cooling fan.However, air flow is better, and constant air flow is more than enough, so for sure you don't even need additional noises introduced by a unit that controls the speed of a bldc motor, and the motor itself needs to sink current from the 12V power-rain, so before and not after the LDO that supplies the SoC.That was a very unusual choice. Not cleaver and rather stupid, and I regret not having noticed this before, but I tend to *trust* the hardware of a product, in fact I spent two months thinking it was all software problems (and there are really problems even with the software).All boards arrived with a bug with the thermal-zone set in the factory u-boot that, worse still, set the dram clock to the highest value possible, the module is sold without any heat-sink, and they say "it's ok, trust us", I tried myself, with this configuration the module fries like fry chips, and if it doesn't fry immediately (which, at 1.2Ghz, is just a matter of time) ... well ... you experience several weird behaviors, and continuous crashes.I too prefer a medium cooling fan to quite operation and sufficient airflow. I have just ordered a dozen 5V ultra silent cooling fans. Just a 5V version of those used years ago on 486-cpus.The parcel is on its way, in the meanwhile, the first thing to do is to remove the on-board connector for the cooling fan, this will prevent my colleagues to install one.

author: Nominal Animal
author_rank: Super Contributor
timestamp: October 12, 2021, 10:42:56 am
content: Quote from: DiTBho on October 12, 2021, 08:15:28 amTrue. I personally might still put a very low RPM (say, 500) 12V PC chassis fan in a pull configuration in the enclosure, through a tight mesh or a filter. The filter helps with dust and the fan noise, too. A suitable 120×120×25mm, say a Nidec Gentle Typhoon, if you can find one, is cheap, basically silent when undervoltaged for lower RPMs, and is well suited for the task. (I haven't used 5V fans, so have no opinion/guidance with those at all.)For some of my SBCs, I've been looking at ways to mill a new lid for a Hammond die-cast aluminium enclosure, to act as both "lid" as well as the primary heatsink. I can get the enclosures cheap locally, and a suitable chunk of aluminium is not that expensive, although most of it will be turned into swarf.. the idea is to have sparse ribs on the outside, and pedestals on the inside to meet the SBC chips that heat up, so it has to be much thicker than just a lid. Wires are pulled through via grommets at the lid-box seam, and the enclosure itself is basically airtight.Quote from: DiTBho on October 12, 2021, 08:15:28 amVery good idea. If one does not want to modify the board itself, I'd recommend finding a suitable connector without wires, and pushing that in. Or even 3D-printing a tiny disabled connector, maybe with a "BAD" or something in raised lettering where the wires usually come out..(The plastic shroud of the fan connector usually comes off just by pulling, but itcouldbe glued to the board so that pulling it might damage the board. These boards are multilayer and so tight I wouldn't want to try to desolder a 2/3-pin TH connector off.)

author: DiTBho
author_rank: Super Contributor
timestamp: October 25, 2021, 11:37:00 am
content: The system is now up and running with an up-time of more than a weekDays ago I developed a monitoring program that runs continuously in the background with a low priority and checks the integrity of each rootfs file(1).Code:typedef struct{char_t filename;md5sum_t md5sum;} mon_entry_t;There is a huge database in ram with the md5 checksum of each file of the rootfs, the monitoring program can calculate a fresh md5-checksum on the fly and compare with the value stored in the database.It's also useful to find corrupted files due to filesystem crash, and this way it's easy to find the corrupted file and selective restore it from a backup.(1) with "rootfs" I mean files in these foldersCode:/bin/etc/lib/opt/sbin/usr/bin/usr/include/usr/lib/usr/libexec/usr/sbin/usr/share/root, /home, /local, etc; services folders such as /var, /dev, /sys, /proc, /tmp, are not considered.The database consumes ~90Mbyte of ram.