[solved] modern gcc pressure - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 12:02:50 am
content: I am experimenting weird behaviors with a SoM.For some silly reasons I found the DDR clock set to 900Mhz while it should be <700Mhz. I cannot fix it at the moment because I don't have access to the firmware file (u-boot) and I am going to fix it next week.Even with the wrong DDR clock speed, the SoM appears *somehow* stable if you do not stress the ram, otherwise the kernel crashes, and this is really something you wouldn't like to see.It seems that some issues happen under heavy load, which is difficult to emulate on u-boot, so to expose the issue I am using cmake -J8 running eight c++ in parallel to compile ClangClang is really an heavy job for the SoM, and I am doing it only for testing reasons, I mean, usually I cross-compile this heavy stuff on a x86 machine.It seems recent gcc versions (>=v10) take 1 GB to 1.5 GB of RAM per job, and if the system has the 8 logical CPUs, but only 512MB RAM (which is insane, but that's the SoM I have here), the MAKEOPTS value should be lowered, this is so that the system has RAM to run the basics as well as compile without hitting swap very often slowing things down.At the moment the kernel has 2Gbyte of swap allocated on an attached USB HDD, and the weird thing is:with Gcc-v10, the kernel crashes ... claiming the DRAM has some issues (1)with Gcc-v8, the kernel doesn't crasheswith Gcc-v7, the kernel doesn't crashesAnd I am perplexed about this(1) which is correct, the DRAM clock is wrong, even if it passes all the memtests(2) on u-boot(2) on u-boot these tests are not as stressing tests as with compiling clang with gcc v10 under Linux

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 12:04:40 am
content: kernel v5.12.0-rc2

author: chickenHeadKnob
author_rank: Super Contributor
timestamp: August 15, 2021, 01:45:12 am
content: It might be as simple as the SOM running hotter (thus slower timing) under one load versus the other. If you are borderline with the timing than any little thing could push you over the edge. I remember a case a long time ago where we had an intermittent and fairly rare crashes on a prototype system. Everyone blamed my software. I couldn't reproduce until I had the idea of heating the board with a heat gun. I then narrowed it to DMA controller timing PAL by accident when I touched the warm board on certain PAL pins adding some capacitance. That allowed the board to function correctly even when warm. I got lucky there.What I am getting at is if you are borderline then I wouldn't place great significance on the C compiler.

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 11:05:08 am
content: Quote from: chickenHeadKnob on August 15, 2021, 01:45:12 amThere is a heat-sink on the top of the SoC chip in the SoM. But the DDR3 ram chip has no heat-sink.Quote from: chickenHeadKnob on August 15, 2021, 01:45:12 amI think it's somehow connected. Gcc v10 tends to consume much more ram, and move more things to the cache.I need a way to test the reliability of all the cpufreq voltage/frequency settings.I am not the author of the hardware and mechanical design, I am the software guy, but I need hardware reliability testsFirst, I have to tell my colleagues the importance of the hardware diagnostic tools

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 01:07:20 pm
content: anyway, it's frustrating that for common things you don't notice the problem. For instance, I have just compiled ruby with Gcc v10, ruby has several parts written in C++ , that tend to stress the ram more than parts written in C, and frankly I was expecting a crash, but no crash.I mean, I would never have noticed the flaw if I hadn't tried to compile llvm with gcc v10 as reliability test.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 15, 2021, 06:05:21 pm
content: There are a number of dedicated programs for RAM testing. You should probably use them for this purpose rather than relying on running programs you know are taxing for memory, but without knowing in what way. I don't know u-boot much, and thus don't know what the "memtests" you mention are. Can you elaborate?memtester on Linux is a classic one if you want to try something directly from a Linux session. It's available as a package on many distributions. If you're using a custom Linux, you can build it from sources:http://pyropus.ca/software/memtester/

author: nctnico
author_rank: Super Contributor
timestamp: August 15, 2021, 09:52:55 pm
content: Fix the memory clock first. Likely the problem isn't the load but the access patterns. A long time ago I had a defective memory module in a computer which caused large compilation runs to fail every now and then. Without the right memory clock the hardware is useless because you can't rely on it so every minute you spend on this is a waste of time. Even if the compiler doesn't crash, it may produce garbage anyway.

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 10:23:04 pm
content: Quote from: nctnico on August 15, 2021, 09:52:55 pmThis is exactly the point of frustration. Imagine if I hadn't tested under heavy load - as this is a commercial product you assume it should work fine, so, shallowly, I would have thought everything was fine.And with gcc-v7, gcc-v8 don't even see any problems. Everything passes internal tests: ruby passes its tests, gcc passes its tests, etc. Only with gcc-v10 and only under heavy load you see the kernel crashing for strange reasons, and only then you start investigating why.This is how I found that the clock setting of the dram in u-boot is completely wrong

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 11:01:57 pm
content: The SoC has 4 cores, each core has 2 threads -> 8 logical CPUs.Quote from: SiliconWizard on August 15, 2021, 06:05:21 pmNothing special, since I don't have any tool on hands, I tried cmake -J8 running eight Gcc-v10/c++ in parallel to compile Clang-v11; the process takes 8 hours and stresses the cache, the ram and the swap.

author: DiTBho
author_rank: Super Contributor
timestamp: August 15, 2021, 11:49:31 pm
content: it's DDR3-1333

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 16, 2021, 12:09:02 am
content: Quote from: DiTBho on August 15, 2021, 11:01:57 pmI'd be curious to see what you get with the memtester program I suggested. It tests memory thoroughly - I've used it in the past to check bad RAM sticks and it spotted the faults with no problem - but it's light on CPU load. So if you let it run for a while and all tests pass, then it might just be a matter of temperature, as chickenHeadKnob suggested. Which wouldn't be surprising, as timings will degrade as temperature rises.

author: DiTBho
author_rank: Super Contributor
timestamp: August 19, 2021, 05:27:34 pm
content: During the last three days I got sources, hacked u-boot and changed a lot of things.Now the EHCI works, I can save the env to the SDcard, a lot of improvements.Anyway, in the binary the I got last week, the DRAM was set to a wrong frequency and Gcc v10 froze as described in previous posts. I set it to different values for lower frequencies and tested the ram with memtest v4.5.1Code:Stuck Address : okRandom Value : okCompare XOR : okCompare SUB : okCompare MUL : okCompare DIV : okCompare OR : okCompare AND : okSequential Increment: okSolid Bits : okBlock Sequential : okCheckerboard : okBit Spread : okBit Flip : okWalking Ones : okWalking Zeroes : ok8-bit Writes : ok16-bit Writes : okThe SoM has an head-sink and a cooling fan, the max monitored temperature was 31 C2x360 -> 720Mhz: crashed after 45 loops2x312 -> 624Mhz: stable after 99 loops

author: DiTBho
author_rank: Super Contributor
timestamp: August 23, 2021, 10:53:59 am
content: With an updated version of Ninja I compiled again llvm and collected the max stack usage during the processGcc-v8: 250M of 2.00GGcc-v10: 1.04G of 2.00G