Is it considered normal to have to periodically reboot a Raspberry Pi? - Page 1
Forum section: electronics / general

author: e100
author_rank: Frequent Contributor
timestamp: February 26, 2022, 04:15:10 am
content: I have a Pi that runs a bunch of programs including Node-red, Grafana, Influxdb, Mosquitto, VNC and my own Java program that runs a home automation system.Generally this runs fine, albeit a bit slowly due to the amount of data being received and logged to SSD.During testing I was rebooting every day and didn't see any problems.Now I run the system continuously and every few weeks the OS kills a bunch of programs presumably because it has run out of memory or some other resource. From what I've seen on the web this is standard Linux behavior designed to prevent the system from completely grinding to a halt. This seems to be a bit different from Windows which tries to keep everything running until the bitter end where everything does grind to a halt.For Linux is the only solution to pre-emptively reboot or kill/restart processes on a schedule so the OS never runs out of resources? I could restart things in the early hours of the morning when activity is at a minimum, but this seems like a drastic measure, and obviously this means there is a window of time where the system isn't doing the job it's supposed to be doing.My example is relatively trivial, but how do important systems such as water, gas or electricity utilities keep their systems functional all the time? Do they have multiple systems running in parallel shadowing each other so they can periodically take one off line to reboot or restart its processes?

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: February 26, 2022, 04:57:30 am
content: To broadly answer your question, the short answer is that the Pi is in a different price class compared to critical industry components and software.Is your problem hardware (interference) or software (bugs)? If you tell us a bit more we may be able to help you lock down what is going wrong.Have you done any logging of resources to try and spot the culprit?

author: BradC
author_rank: Super Contributor
timestamp: February 26, 2022, 05:44:53 am
content: Quote from: e100 on February 26, 2022, 04:15:10 amThe solution is to find and fix the memory leak. I have pi's with hundreds of days uptime.

author: james_s
author_rank: Super Contributor
timestamp: February 26, 2022, 07:01:54 am
content: I have 3 of them in service 24/7, one had an uptime of 280 days when I tripped over the power cord, it's currently up to 266 days. Another was rebooted about 4 months ago when I last updated Home Assistant. The third probably gets rebooted 2-3 times a year when Kodi gets updated. If you use a decent quality SD card and an adequate power supply a Pi should be very stable.

author: wilfred
author_rank: Super Contributor
timestamp: February 26, 2022, 07:44:18 am
content: It doesn't sound like normal Linux behavior to me. And you saypresumablybut presuming won't get you anywhere. You need to try to gather some data. It may not even be a memory issue.It could be faulty hardware or a marginal power supply or a flakey SD card. Maybe one of your applications does have a memory leak or some other bug. You need to do more detective work.But Linux ought to be able to run without rebooting for ages.

author: Whales
author_rank: Super Contributor
timestamp: February 26, 2022, 07:44:36 am
content: Quote from: e100 on February 26, 2022, 04:15:10 amThis is the "out of memory" or "OOM" killer. You can configure it if you want it to behave differently.If you add swap space: the system will start "grinding to a halt" instead, as it tries to swap RAM to disk and does its best to keep going.Overall however: you need to find out what your hog is. A misbehaving application can easily (try) to eat TB of RAM, so even the best system in the world won't handle it for long. It's impossible to write a fair OOM killer algorithm as applications can cheat by spawning lots of threads/processes or hogging more obscure/complex resources.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 26, 2022, 07:54:55 am
content: It sounds like at least one of your userspace processes consumes increasing amounts of RAM, until the kernel Out-Of-Memory killer is triggered. This is expected and as-designed behaviour.(Although the 'Pis are not the best SBCs out there, the only hardware issue I know of is of the"in certain, very specific circumstances, may lose data"sort, and not of the crashing sort. Running the same software on any similar SBC with the same amount of RAM, you'd have the exact same issue. If it was a power supply issue or similar, usually the Pi becomes completely unresponsive. The fact that only some processes are killed, is a telltale sign of the OOM killer. The kernel logs,dmesgor those in/var/log/, will tell exactly what happened.)What you can do, is log e.g.ps axfuww | sort -grk 5to a file once every hour or so (using a script in/etc/cron.hourly/), and find out which process that is; that command lists the biggest memory hogs first. (The logs will tell you which processes the OOM killer killed, but only logging all processes will tell us exactly why it happened. A single log is not useful either; you want to know which process actually grew continuously.)When you find the process that grows overmuch, and you run it as a service, you can use say a daily script to restart that service, until you fix the memory leak.Otherwise, you can use a daily script that uses e.g.kill -HUP $(pidofprocessname)orkillall -HUPprocessnameto send the memory-hogging process a hangup signal (-TERMand-KILLare alternatives; you can even use the HUP as a gentle reminder, TERM as a hard request, and KILL as no-questions-asked signals with a couple of seconds grace time in between), and then re-start the process.This all depends on first determining which process or processes do leak memory, though.For future reference:You can set process limits using e.g.prlimitfor each individual process (and their children) when you start them. Instead of/path/to/binaryarguments...have your script run e.g./usr/bin/prlimit --as=64M:64M /path/to/binaryarguments...This means that the process cannot exceed the given limit. Theasoption limits the size of the address space in use by the process. If it tries to exceed that, syscalls likesbrk()andmmap()will fail; in practice, it means that memory allocation will fail.Similarly, for processes that are less important than others, you can usenice -nlevel(process priority) and/orionice -nlevelto reduce their priorities. For nice, the level is 0 (default) to 20 (least important), and for ionice, 0 (default) to 7 (least important). Just chain them like prlimit above.You can even "manually" change the OOM score of the process (identified by its process ID number, and controlled by/proc/PID/oom_adjand/proc/PID/oom_score_adj, or/proc/PID/oom_score), so that if OOM occurs, the OOM killer targets or avoids (based on the adjustment or score) that particular process. Seeman 5 procfor the descriptions of these pseudofiles.Many people often complain about the Linux OOM killer, without realizing that it simply applies a policy that us humans can trivially set in userspace. The default policy is a compromise between all sorts of use cases, intended to be "do the least harm in any situation", with a heuristic on how it targets the processes it kills. This has actually been a computer science research topic for a couple of decades at least, and nobody has found a better automatic heuristic on choosing what to kill; itsoundslike a simple problem, but when you realize that different types of services and applications naturally use different amounts of resources, and their importance varies depending on what the use for the device is, it becomes intractable. The only sane solution is to use minimally invasive defaults, and let human administrators tune the settings as needed, if needed. Usually, it's not needed.Process limits are a separate mechanism. You can limit even how much CPU time a process can stay running. Memory limits affect how much memory the process can request from the kernel. These all have a soft limit, and a hard limit. For example, when the process exceeds the soft CPU time limit by one second, it gets sent an SIGXCPU signal. If the process then exceeds the hard CPU time limit, it is killed via SIGKILL signal, which cannot be blocked or stopped at all.Since different workloads are usually groups of processes and not single processes, you can utilize control groups AKAcgroups. For typical small appliance use cases, you don't need those; it's when you have multiple concurrent workloads that need different limits so they don't unduly affect each other, that cgroups become most useful.Some Linux SBCs, but not Pis as far as I know, use ARM big.LITTLE architecture, where the same System-On-Chip has both "big"/fast CPU cores and "small"/slow CPU cores. For these, thetasksetcommand can be used like the resource use limiting commands above, to specify which CPUs the process (and its child processes) can use. Combined with nice and ionice, this allows restricting a process or a set of processes to run only on the "small" (or "big"!) cores.

author: Whales
author_rank: Super Contributor
timestamp: February 26, 2022, 07:55:35 am
content: In terms of SBC reliability: my biggest problems have been wifi related.Raspberry pi Zero W with Alpine linux: wifi sometimes reconnects but then doesn't work. I think it might have come down to Alpine's udhcpd not getting a new ip lease. Not sure, worked around it by writing a watchdog that did a "service network restart" if things failed. Happened every few weeks.GL-MT300N-V2 (yellow mini router) with Openwrt: wifi drivers sometimes failed spectacularly, dmesg showed all sorts of goof. Solution was writing a watchdog to reboot the units. Happened every few weeks.Otherwise I have great luck with embedded linux and reliability. Some of my OpenWRT devices have run for over a year (the only downtime is for flashing updates); and they run in a much more constrained resource environment than your raspis, albeit all of their software is chosen for reliability in such environments.

author: BradC
author_rank: Super Contributor
timestamp: February 26, 2022, 08:19:33 am
content: Not a raspberry pi, but mac mini :brad@xxxxx:~$ uptime05:37:43 up 1716 days, 3:07, 2 users, load average: 0.89, 0.36, 0.17

author: janoc
author_rank: Super Contributor
timestamp: February 26, 2022, 09:10:56 am
content: Quote from: e100 on February 26, 2022, 04:15:10 amI am pretty sure your code is leaking memory. Stuff like Grafana, Nodejs, InfluxDB etc. isn't designed to run on machines with low amount of RAM and that have to stay up for weeks/months. Leaking a megabyte of RAM here and there on a PC or a server instance somewhere in AWS pretty much doesn't matter - but will reliably make a small machine run out of memory. The bigger one would too, eventually, but they usually don't run for that long for this to become a problem these days ...Also, throwing an exception on error and dying is a common error handling "pattern" - few if any of this software is written with any sort of resilience in mind.Quote from: e100 on February 26, 2022, 04:15:10 amWell, critical stuff that must stay up doesn't run nodejs and Grafana on a RaspberryPi, for starters. Such things can be used - but only for non-critical systems, such as analytics or some part of monitoring. That is what it has been designed for. So if your Grafana dashboard goes down, the CEO will be pissed because they can't see their reports but the water won't stop flowing and the country won't go into a blackout.Concerning running "mission critical" stuff that must stay up, there are multiple strategies and not everything is always applicable. You need to evaluate your tradeoffs between how resilient the system you want to be and how much it will cost.Common solutions in no particular order:- Service monitoring with automatic restarts - e.g. systemd can be configured to monitor your service (check if a process exists) and restart it if it goes down. If you don't like systemd (which likely comes with your RPi distro), there are other such solutions too, even more sophisticated, e.g. Monit that can be configured to do rather complex monitoring - e.g. not only check whether a process is running but also whether the service is responding to requests and how quickly.- Logging information, continuously analyzing the logs, identifying andfixingthe reasons for the failures. Restarts are only a band-aid to get your out of trouble right now, they don't solve the underlying problem that could cause you to lose data if/when the service crashes, for example.- Redundancy - multiple copies of the critical service with a load balancer/automatic failover. If one instance goes down, others will pick up slack transparently. Same can be done on hardware level - multiple servers, configured with dynamic failover so e.g. if the server itself or its network connectivity dies, the service remains running. This is often done with having the servers in geographically spread out locations, so that e.g. a power outage, flood or fire in one datacenter doesn't take out everything.- Redundancy 2 - use multiple copies of the service simultaneously and constantly compare the results. If they don't match, eliminate the misbehaving copy. This is very common in safety-critical systems such as airplane control systems where you have 3 computers running the same* software in parallel and there is a voting/comparator system that checks their outputs. If two match and third doesn't, it is voted out and marked as failed. If you have only 2 computers, then if their outputs don't match, the system detects it and fails both, typically handing control to a human operator to deal with the problem.Famously, B737 MAX did not work like this - they have only 2 computers because it is not a fly-by-wire plane, so even if the automation goes on a fritz completely the pilots can still control it and such high redundancy level isn't required. Each of these was fed data from one of the angle of attack sensors. Unfortunately, the system did not actually use both sensors simultaneously, constantly comparing them but only one has been used at a time. So there was essentially no redundancy at all.In the previous versions of 737 this wouldn't have been a major problem (failed sensor would only trigger some alarms in the cockpit and some automation would stop working, the computers were controlling the "nice to have but not safety critical stuff") but in the MAX there was this new added system that actually affected controls of the plane and pushed it into a nosedive because of the bad data. That's what you get when you bolt on safety-critical functionality (without realizing) on something that hasn't been designed to be safety critical from the start ...- Safe coding practices - if you don't allocate memory at runtime, you can't leak it either. This is why stuff like nodejs (or Python) is not something you will find in mission-critical applications because you have no control over how the application manages memory. See also:https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Developing_Safety-Critical_Code(used by JPL/NASA). There is also the concept of "defensive programming" - identify what could go wrong, evaluate the risk - and mitigate it right when you are implementing the functionality. It ismuch easierat that time vs having to deal with a behemoth of an application with millions of lines of code crashing due to invalid input somewhere.- Architecture - split the application into a small, safety/mission critical core that will be very carefully written and tested/verified. All the optional, non-critical "fluff" like UIs should be separate processes/components that can be talking to the critical core using some well defined interface. The idea is to isolate one from the other so that your car or rocket don't blow up only because a control panel or infotainment display has crashed. Of course, the mission critical part needs to be able to identify and handle any erroneous inputs/data any such crashing UI or sensor could send it too, otherwise you are back to square one. Quite a few space missions were lost because the code was not handling invalid data correctly. Also the 737 MAX crashes started with exactly that - the on board computers being unable to identify and deal with invalid data from a failed critical sensor. Again, this goes back to the safe coding practices.*I wrote "running the same software" but usually you don't run exactly identical copies. The idea is to have at least one "known good" older version running on one of the systems, so that in case your current release has introduced some kind of undetected bug, it gets caught. If you had all computers running the same faulty software, they would all fail in the same way with the same input (computers are deterministic) - and that could be pretty catastrophic. The Space Shuttle flight computers have been designed and used like this, for example. Also the Airbus fly-by-wire control computers are working like this.

author: Microdoser
author_rank: Frequent Contributor
timestamp: February 26, 2022, 10:11:04 am
content: I have a similar issue with running some code on a Pi.A workaround I am using is to use psutil:https://psutil.readthedocs.io/en/latest/It allows you to check available memory against total memory and I have put in code to reboot the system when a threshold is reached. The Pi then autostarts the required application.This should allow you to have the system running, albeit with occasional breaks in monitoring while the Pi reboots, while you find the memory leak.Is the monitoring of such time sensitivity that a 30-60 second gap is critical?

author: DiTBho
author_rank: Super Contributor
timestamp: February 26, 2022, 10:43:47 am
content: I understand you. Last summer I got an H3 SoC and it randomly crashed every 24h.I then spent several months fixing defects of both firmware and kernel and encountered several errors there. For example, I found that u-boot was using the wrong timing, the kernel was initialized with the wrong device tree values, and the thermal management of the kernel was just plain buggy and not correctly functioning. I removed it, I fixed my DTS file and got both the serial ports and the two USB ports working, I re-implemented things in u-boot (and found there were also Endian-issues with the disk-partition), but one thing above all these mess up, there was also a stupid hardware bug: the cooling fan was connected directly after the LDO and this it was causing current glitches which caused abnormal behaviors in the SoC, which in my case cause catastrophic filesystem corruption.There should be my topic here on EEVblog where I talked about that.In conclusion, I spent just 30 euro for the board and four months to make it working. Linux on these cheap devices is like that: it makes you *want to do it* because it is cheap and you see that people who talk well about it on Youtube, but when you put their hands on it ... you have to beat yourself up a lot with wild hacking things.

author: emece67
author_rank: Frequent Contributor
timestamp: February 26, 2022, 11:09:01 am
content: .

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 26, 2022, 12:08:54 pm
content: Quote from: emece67 on February 26, 2022, 11:09:01 amStart with the SoC manufacturer. Broadcom and Allwinner aren't exactly forthcoming with their chipsets, so their Linux support is obviously on shaky ground from the get go. Even if you get a stable kernel version from somewhere, their support isn't in mainline kernels, so you'll be beholden to whoever forked the kernel for that version. Not good, in the long term.Amlogic S805 and S905 (Meson family) has pretty good support in the mainline kernels; you're not beholden to the vendor for support. The features implemented on the SBCs are described by Device Tree, so you'll also want to find out if the exact board has Device Tree descriptions upstream. Although anyone can construct the device tree source (and compile to a .dtb), discovering the exact addresses and connectivity et cetera is quite a lot of work.These are extremely common in Android-based TV boxes; I got one, H96 MAX X3, but its DT support in Linux isn't complete yet.Samsung Exynos SoCs are alsosupportedin mainline kernels, but DT descriptions exist really only for the most popular boards (Odroid XU4 and HC1/HC2, I'm not sure of others).You can find the list of Device Tree descriptions for all 32-bit ARM-based Linux SBCs in the mainline kernelshere, and for all 64-bit ARMhere. The Device Tree Source format is human-readable; see theOdroid HC1or sayLibre Computer's La Fritefor example. Note that a.dtsfile can include another; the common includes are usually named.dtsi.One key feature you want, is that the SBC must be able to boot from either built-in Flash (where e.g. u-boot will reside), eMMC or dedicated boot Flash, or from external storage. You do not want to rely on microSD cards for booting, if reliability is something you want. (If you can find a good, reliable microSD and keep it mounted read-only, then it's okay; but it is darned hard today to determinewhichmicroSD cards are reliable in this kind of use in the long term.)All SBCs with an USB 3 port can use SATA SSD disks via cheap USB-SATA bridges. Not all bridges are equal, but only connecting one and looking at the Manufacturer:Product number (hhhh:hhhh in hexadecimal) will tell for sure; just runlsusb. The manufacturers often change implementations without changing anything outwards visible, which frustrates us Linux folks when the earlier chip used in the USB-SATA bridge has public datasheets and is supported in Linux, but the new one doesn't. Very few SBCs, however, can load the bootloader off USB, so you definitely want some kind of boot flash on the SBC.All this is just my suggestion – I have less than a dozen different SBCs – and others may disagree or have better anecdotal evidence.

author: rsjsouza
author_rank: Super Contributor
timestamp: February 26, 2022, 02:04:16 pm
content: IME the Beaglebones, although not full fledged SBCs, work reliably over long uptimes. Sure, the majority of issues listed here are related to the applications themselves and their achilles' heels, so the foundation has to be solid as well (I got burnt by too many low level bugs in my lifetime so I now limit my options w.r.t. reliability).And yes, Linux can be an expensive solution.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 26, 2022, 02:16:21 pm
content: The process:For example usingtop,psorlsof, look at which process hogs significant part of available system memory, CPU time, file descriptors, or other limited resources.External software -> replace with something non-broken. Use well-known, well-tested good software.Your own software -> debug and fix the memory leak.Use auto-restarts and similar only when in absolute panic mode, and only temporarily. Otherwise, fix the bug. If it seems overwhelming, remove excess parts until the problem is simple. Home automation should not be too complicated. Don't throw more software at the problem; reduce instead.

author: DiTBho
author_rank: Super Contributor
timestamp: February 26, 2022, 02:50:46 pm
content: @Nominal Animalnow I am with aMediaTek's SoC. Don't remember if I have ever talked about the "Moka project". It's a weird things of mine where the GPU Mali is disabled and you only work with the ARM cores of theMTK MT7622. Kernel >= v5.10 + BSP.It's called "Moka" because it seems that you need a lot of coffee to make things working with these kind of SBCs.We will see how it goes

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 26, 2022, 03:26:09 pm
content: Quote from: DiTBho on February 26, 2022, 02:50:46 pmSounds interesting. I was surprised to find thatMT7622is a dual-core ARM Cortex-A53, as MT7621A (in the Mikrotik RBM33G I have) is a MIPS 1004KEc.I don't know what to really think of Mediatek SoCs. On one hand, it is used inmany routers, some of which are quite stable with OpenWRT. On the other hand, Mediatek tends to just fork a kernel, stuff some kind of driver in there (with the code being... well, not something anyone would show in their portfolio, let's say), and that's it. It's better than Broadcom or Allwinner, as at least there is something one can read in order to write proper support drivers, but nothing like Amlogic, Rockchip, or Samsung efforts: they have actual developers submitting support patches to the mainline kernel, so that everyone using their SoCs benefit long-term.Oh yeah: Rockchip is one of the very rare Chinese firms that do try to push the support of their SoCs into mainline kernelsproperly. Commendable. I'm seriously considering getting theOlimex RK3328 SOM and carrier, but as of now, have no real-world experience on the chips. (The kernels Olimex provides are not vanilla, but Rockchip's own, with patches and support that may not have yet been pushed to the mainline kernels.) I've just noted their paid developers' activities in the Linux kernel developer mailing lists; they have my respect thus far. (Olimex has a long-time commitment to OSHW, too.)Quote from: DiTBho on February 26, 2022, 02:50:46 pmWhatever you do, do not check what it means inFinnish.

author: DiTBho
author_rank: Super Contributor
timestamp: February 26, 2022, 04:34:39 pm
content: LOL, there are several ways to prepare coffee in the world:Moka (the one shown in the attached pic)Chemex (the bulgarian girl in my group is the only one who uses it)Pour OverFrench pressSoviet press... (probably more)I know because in my group of friends none of us use the same method.My preferred one is "Moka", also knows as "cafetera de fuego" (Spanish) and it's the name of a coffee maker whose etymology comes from "Mokha", a city of Yemen, in the Arabian peninsula.

author: DiTBho
author_rank: Super Contributor
timestamp: February 26, 2022, 04:50:17 pm
content: Quote from: Nominal Animal on February 26, 2022, 03:26:09 pmI am still supporting the RB532A because it's used in a very complex project. However, I still don't have a good plan for Catalyst supporting MIPS32/LE, I mean ... I still only have support for MIPS32/BE.No updated Stage3s. The last one is 2017. Not too bad, but it's 5 years old stuff and I seriously need to cook a new one.Sooner or later I'll * copy * you and buy a Mikrotik RBM33G, so I'll have no excuse for being so lazyQuote from: Nominal Animal on February 26, 2022, 03:26:09 pmYup, Mediatek seems several light years ahead than Allwinner. At least u-boot comes with the right settings and I don't have to spend time on it. The kernel support also looks better, even if ... on the miniPCIe there are a lot of quirks and the GPU is something that needs more work.That's why I have disabled it for now, also because I seriously need the miniPCIe in perfect working order. There is a fun (and rather crazy) plan for it. (here I think you might guess what I mean)p.s.According to the project plan, the entire rootfs should boot from SATA to CF. But we'll see how it goes.

author: e100
author_rank: Frequent Contributor
timestamp: February 27, 2022, 05:28:04 am
content: Thanks for all the advice. The Pi is running a 32 bit OS on a Pi4 with 8GB memory and is running from a USB connected SSD, not an SD card, so the first thing I did was to increase the size of the swap file from the default 0.1 GB to 2 GB. I'll keep an eye on the usage to see if it creeps up beyond the 0.1 GB it was using previously and will use that as my early warning sign of trouble ahead.The first few lines of 'top' looks like this:top - 13:13:42 up 32 min, 1 user, load average: 3.82, 3.21, 2.55Tasks: 172 total, 1 running, 171 sleeping, 0 stopped, 0 zombie%Cpu(s): 16.0 us, 9.4 sy, 0.0 ni, 70.9 id, 2.6 wa, 0.0 hi, 1.0 si, 0.0 stMiB Mem : 7897.7 total, 4627.5 free, 1263.3 used, 2006.8 buff/cacheMiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 6068.6 avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND563 influxdb 20 0 2487652 135944 31604 S 23.2 1.7 5:04.80 influxd455 pi 20 0 229880 116548 25476 S 17.2 1.4 5:47.92 node-red110 root 20 0 137256 88832 87836 S 12.3 1.1 3:57.45 systemd-journal1274 pi 20 0 932848 11308 5440 S 9.3 0.1 2:47.63 nocand753 pi 20 0 129728 49192 18148 S 8.9 0.6 2:52.87 Xvnc-core1649 pi 20 0 1610660 653552 28304 S 7.9 8.1 8:18.24 java1279 pi 20 0 802456 13588 5736 S 7.3 0.2 1:42.57 nocanc2416 pi 20 0 1200664 111596 15308 S 7.0 1.4 2:40.28 java421 root 20 0 25512 3388 2436 S 6.0 0.0 1:49.81 rsyslogd556 mosquit+ 20 0 8900 5252 4696 S 2.6 0.1 0:51.85 mosquitto91 root 20 0 0 0 0 S 1.0 0.0 0:20.24 jbd2/sda2-86138 pi 20 0 10428 2856 2464 R 1.0 0.0 0:00.05 top919 pi 20 0 428528 33268 26032 S 0.7 0.4 0:16.87 lxpanel

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 27, 2022, 07:06:34 am
content: FWIW, influxd has a history of memory leaks. (Just do a search on"influxd" "memory leak"to see for yourself; re-occurs in different versions.)This type of memory leak will most likely show up as slowly increasing VIRT size, i.e. larger and larger virtual memory address space being used, because it never returns any back to the OS. You can usetop -o VIRTto sort the list so that processes with the largest VIRT are shown first.

author: Siwastaja
author_rank: Super Contributor
timestamp: February 27, 2022, 07:35:06 am
content: Thisshouldbe obvious, but you just can't have memory leaks in software supposed to run 24/7, server software, daemons, services, etc. It's just... total no-go. Even if they run in userspace, the same scrutiny as in kernel development is needed; in the end, it doesn't matter whether the kernel crashed or if an important userspace daemon crashed, the end result is non-functional system.Memory leaks can be accepted in experimental or otherwise interesting desktop software where reliability or ability to run non-stop is uninteresting, because you can forgive the developers some mistakes, seeing they are human beings, too; that is, if the software is otherwise useful. But a database service just can't have any memory leaks. If it needs autorestarting on regular basis, I just can't see any use for it.

author: DiTBho
author_rank: Super Contributor
timestamp: February 27, 2022, 10:48:40 am
content: Quote from: e100 on February 27, 2022, 05:28:04 am32bit userspace, 64bit kernel to address 8GB of ram, and do you also need swap?that's worrying for me

author: DiTBho
author_rank: Super Contributor
timestamp: February 27, 2022, 10:54:37 am
content: Quote from: Nominal Animal on February 27, 2022, 07:06:34 amis it the one written in Go-lang?

author: e100
author_rank: Frequent Contributor
timestamp: February 27, 2022, 12:15:39 pm
content: Quote from: DiTBho on February 27, 2022, 10:48:40 amAfter 7 hours it looks like this, so its using the swap for something.top - 20:13:21 up 7:31, 1 user, load average: 3.00, 2.50, 2.29Tasks: 170 total, 2 running, 168 sleeping, 0 stopped, 0 zombie%Cpu(s): 17.0 us, 8.3 sy, 0.0 ni, 63.7 id, 10.0 wa, 0.0 hi, 1.1 si, 0.0 stMiB Mem : 7897.7 total, 3909.6 free, 1249.8 used, 2738.3 buff/cacheMiB Swap: 2048.0 total, 1878.7 free, 169.3 used. 6140.3 avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND563 influxdb 20 0 2492400 235576 30532 S 20.2 2.9 71:16.02 influxd455 pi 20 0 241948 116048 25280 R 16.6 1.4 75:11.47 node-red110 root 20 0 206744 135840 135440 S 12.6 1.7 58:32.73 systemd-journal753 pi 20 0 129728 34004 18032 S 10.3 0.4 35:38.03 Xvnc-core1274 pi 20 0 1088572 7056 3436 S 8.6 0.1 42:00.59 nocand1649 pi 20 0 1628444 654756 26736 S 8.6 8.1 40:18.44 java1279 pi 20 0 802456 11040 3476 S 8.3 0.1 24:43.13 nocanc2416 pi 20 0 1203752 115516 14708 S 7.0 1.4 34:02.90 java421 root 20 0 25512 2740 2056 S 5.6 0.0 27:07.41 rsyslogd556 mosquit+ 20 0 9412 5324 4696 S 2.3 0.1 12:46.54 mosquitto1333 pi 20 0 236940 36752 34284 S 2.0 0.5 0:57.60 lxterminal

author: BradC
author_rank: Super Contributor
timestamp: February 27, 2022, 01:13:04 pm
content: That's normal. Linux will drop stuff into swap to maximise available buffer space.Even with obscene amounts of free memory.Code:brad@srv:~$ free -mtotal used free shared buff/cache availableMem: 64319 21323 570 6 42425 42281Swap: 40927 4168 36759

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 27, 2022, 01:25:09 pm
content: e100, note that your list is sorted by CPU load (default), and unless you are actually looking at the entire listing, this initial part of the list may not even contain the true memory hog. (As BradC said, a bit of swap used is completely normal and expected.)I recommend you pressfto get to theField management,then Up and Down keys to get to theDATArow,then pressdto display it, andsto sort the output by that,then Up and Down keys to get to theUSEDrow,then pressdto display that too,and finally pressqto get back to the listing.This will show DATA and USED columns (at the right edge of your terminal window), sorted by DATA.Quote from: DiTBho on February 27, 2022, 10:54:37 amNo idea. It's just that the resident set size (RES) compared to the address space allocated (VIRT) made me do a quick web search, on"memory leak" "influxd".

author: e100
author_rank: Frequent Contributor
timestamp: February 27, 2022, 03:30:15 pm
content: I'm guessing that the top one (1649) is the Intellij IDE and 2416 is my Java program running inside it. Don't know what 2399 is, perhaps it's something to do with Intellij."nocand" is fromhttps://github.com/omzlo/nocandand allows you to send/receive MQTT sized messages over a CAN bus and do over-the-wire firmware updates. Unfortunately chipageddon has forced the developers to focus on other stuff.PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND DATA USED1649 pi 20 0 1629320 509496 26480 S 5.9 6.3 50:25.65 java 756648 713344563 influxdb 20 0 2493636 389108 63264 S 10.9 4.8 118:12.30 influxd 340372 3915441274 pi 20 0 1104964 7948 3428 S 9.5 0.1 54:08.60 nocand 307728 8524455 pi 20 0 240416 108100 18916 R 17.8 1.3 97:07.55 node-red 178788 122640552 grafana 20 0 972204 82496 49288 S 0.3 1.0 34:13.73 grafana-server 166480 869002416 pi 20 0 1203752 115100 14288 S 7.9 1.4 43:31.45 java 134428 1151002399 pi 20 0 863092 103436 13640 S 0.0 1.3 0:45.97 java 129480 1034361333 pi 20 0 236940 37712 34700 S 0.3 0.5 1:13.61 lxterminal 109004 46600919 pi 20 0 430520 28036 23684 S 0.7 0.3 4:40.01 lxpanel 74984 32884923 pi 20 0 161196 24240 23520 S 0.0 0.3 0:02.49 pcmanfm 55652 29388422 root 20 0 65188 7304 6952 S 0.0 0.1 0:00.33 udisksd 43256 8840900 pi 20 0 56764 5888 5888 S 0.0 0.1 0:00.04 gvfsd-fuse 42404 66401110 pi 20 0 55112 7300 7272 S 0.0 0.1 0:00.08 gvfsd-trash 33936 81241084 pi 20 0 56028 6268 6268 S 0.0 0.1 0:00.02 gvfs-afc-volume 33932 7080753 pi 20 0 129728 36300 17836 S 6.6 0.4 47:21.38 Xvnc-core 33712 48880

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 27, 2022, 03:47:22 pm
content: Presscto toggle between full command line or just the command name.Quote from: e100 on February 27, 2022, 03:30:15 pmThey look like typical Java processes to me. Keep an eye on nocand, if any of its VIRT, RES, DATA, USED drift upwards.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 27, 2022, 05:35:26 pm
content: I've rarely seen a Java program running for an extended period of time (days, weeks) without leaking significant memory. But maybe I'm just very unlucky.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 27, 2022, 05:44:06 pm
content: Quote from: SiliconWizard on February 27, 2022, 05:35:26 pmThey do need sufficient address space (memory reserved) for the garbage collector to work. And it won't trigger until there is significant amount of stuff to collect, so you cannot run Java stuff with very limited memory at all, in my opinion.I have run black-box Java stuff (with process limits, as a dedicated user, with restricted privileges) reverse-proxied from Apache for years (say 2005-2015 timeframe). I didn't like it, but it did work.

author: DiTBho
author_rank: Super Contributor
timestamp: February 27, 2022, 05:52:27 pm
content: I don't know, guys, but I know Java v1 was a different thing. I have a weather station written in Java v1, it's literally outside the door and no one has touched it for the past 10 years. I have a couple of other applications written in Java v2 and they crash even if you don't touch them.

author: DiTBho
author_rank: Super Contributor
timestamp: February 27, 2022, 05:54:32 pm
content: Quote from: e100 on February 27, 2022, 03:30:15 pmWhich hardware interface do you use for CAN bus?

author: brucehoult
author_rank: Super Contributor
timestamp: February 28, 2022, 12:41:27 am
content: Quote from: BradC on February 26, 2022, 08:19:33 amHere's my always-there nano host at EC2 (I bring up bigger ones when I need them):Code:Last login: Fri Nov 12 01:47:31 2021 from 218-101-100-69.dsl.dyn.ihug.co.nzubuntu@ip-172-31-25-85:~$ uptime00:38:07 up 1398 days, 5:21, 1 user, load average: 0.00, 0.01, 0.05ubuntu@ip-172-31-25-85:~$ top -n1 | head 4Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem: 500212 total, 370440 used, 129772 free, 38020 buffersKiB Swap: 0 total, 0 used, 0 free. 241108 cached MemLinux is stable. It doesn't matter whether you're running on x86 or on a Pi.The applications you are running might not be stable. That's your problem to address.

author: e100
author_rank: Frequent Contributor
timestamp: February 28, 2022, 01:44:50 am
content: Quote from: DiTBho on February 27, 2022, 05:54:32 pmSTM32F042 chips provide the CAN transport layer, act as message brokers and programmers for co-located SAMD21 chips that run your code.https://omzlo.com/articles/the-nocan-platformThe cool feature is that the SAMD21 chips are un-brickable as they are passive players in the firmware update process.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 28, 2022, 04:10:36 am
content: Quote from: brucehoult on February 28, 2022, 12:41:27 amYeah. Actually, this is exactly what the symptom shows (the OOM killer doing its job just to keep the OS afloat.)Of course, that could come from any application that the OP uses, not necessarily what their wrote themselves, but if they are mostly using "standard stuff", chances are, the OP's own app(s) are the culprit.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 28, 2022, 07:03:31 am
content: Quote from: brucehoult on February 28, 2022, 12:41:27 amTrue. There is one class of exceptions, though: bad drivers.There are several classes of "bad" drivers, from bit-rotted unmaintained ones, to new and untested drivers, to out of tree drivers put together with spit and prayers.The Linux kernel is monolithic in the sense that all of it, including drivers, runs in a single completely privileged mode, without any boundaries between its parts. (It is non-monolithic in the sense that itsstructureis quite modular, although it does not have any internal stable interfaces. The only stable interfaces are to userspace applications: things like syscalls, and /sys and /proc pseudofilesystems.) Because of this, a bad driver can make the kernel crash in a completely different part of the kernel.This is also why Linux developers do not like proprietary drivers. It is not really a philosophical argument – the kernel devs are pretty darn practical bunch –, it is the fact that without exact sources, the proprietary drivers are a black box, and because of how the Linux kernel is monolithic, such a driver can do absolutely anything, even override completely different drivers' data.Thatis why using such drivers "taints" the kernel: it is not an ideological thing, but a practical thing, because when such a kernel crashes, it is impossible to fully explore the chain of events that resulted in the crash. It's like telling police investigating a murder they don't get to interview someone who was armed and at the location when the murder happened: they won't be happy, because they can't do their job properly. Sure, the human user can becertainthat proprietary driver had nothing to do with the issue.. and yet, the full chain of events and part of the crime scene is closed to the Linux developers.This is the reason why I recommend using Linux SBCs and devices whose support is in mainline kernels, instead of SDKs or forked kernels provided by the vendor. There are situations where it isn't optimal –– hardware issues for examplealwaysrequire some help from the hardware vendor –– but in my experience, it leads to more stable and reliable Linux appliances.

author: DiTBho
author_rank: Super Contributor
timestamp: February 28, 2022, 08:40:09 am
content: @Nominal Animalyup, I agree.Allwinner has something similar for the "thermal support" of H3 and H5 SoC, and you should have to trust a binary blob loaded from the userspace into the kernel space. No dice! They can forget it!Anyway, the second class of exception is:non-mainstream platforms.e.g. Linux on MIPS-IV/UMA is generally unstable, experimental, and prone to crashe.g. Linux on HPPA2 is unstable, experimental, and you have to be careful what you do because it is easy to crash. You plug in a PCI-sATA card which usually works perfectly fine in an x86 workstation and which you have no doubts about the quality of the driver, but, as PCI support sucks in HPPA2, you get unexpected behaviors (mainly due to cache bug) with catastrophic silently corruption of the filesystem.But Arm@PI is mainstream, followed by a lot of people, and with Vanilla support.

author: Nominal Animal
author_rank: Super Contributor
timestamp: February 28, 2022, 10:19:20 am
content: That's a good example of exactly why the true source of the problem can be something different that seems to crash.In the userspace case, something similar can occur when applications are connected to abus, connected to each other using sockets. When an application starts to malfunction, it may not be a bug in itself, but for example a contract violation in how the bus is supposed to operate. Lack of documentation, lack of description on how such services and buses are intended to operate, is a big problem in today's Linux systems.Memory leaks are usually localized to an application or a library. Sometimes they are easy to pinpoint, but usually they only occur in specific circumstances, when the code path that gets used (because of settings, conditions, et cetera) is slightly different than the developer expected, and malfunctions in that case.For example, when you use C code that relies on the Unixselect()interface, it is easy to forget that it is limited to nonnegative descriptors less thanFD_SETSIZE. In the olden times, that was a limit you'd never reach in practice, because a single process was not allowed to have that many open file descriptors, but on my system, that's just the default soft limit, easily increased if needed. The fix is to switch topoll()(orppoll()orepoll()), but it isn't a trivial drop-in replacement, and some refactoring is needed.Now, the way that bug manifests itself, can be really odd. In glibc,FD_SET(FD_SETSIZE+n, &set)is a buffer overrun for all nonnegativen, and since the set is usually a local variable, can scribble over other variables in the same scope or even the return address. Or it can just show up as some of the clients never getting a response from that service (because the connection got a descriptor number too large).If the code is written in a defensive style, you'd see a check after anaccept()call, something likeif (fd >= FD_SETSIZE), but for whatever reason, that kind of defensive programming is seen as wasteful/unnecessary/paranoid, or just uncool among the hip. (I have often gotten the comment"that check is unnecessary, because if it is ever true, you have much bigger problems already", as if that makes checking for errors somehowlessuseful or less necessary.)Even when fixing such bugs, the first step is always to gather information on exactly what happens –– not just in a small zone one might think ought to delineate the problem, butoverallin the system; because as shown above, the cause of the problem is not where one sees its results appear at.topandpsare very useful tools here. When a process misbehaves in Linux, the kernel pseudofiles for that process in/proc/PID/can be very useful, especiallymaps and smaps(which describe the exact address ranges and their use in that process).Typically, the bugfixing part cannot start before users have located and identified the bug. The easier it is to repeat/duplicate, the better. Memory leaks are annoying to work on unless you're very familiar with the code (because it involves understanding how the memory resources are intended to be tracked; the bug being a failure in implementing that), but knowing exactly the situation when it happens, would help the programmers find the misbehaving code. I say this from experience.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: March 02, 2022, 07:05:06 am
content: One of my Pi’s was last rebooted in January 2019.

author: james_s
author_rank: Super Contributor
timestamp: March 05, 2022, 07:01:36 pm
content: Quote from: Sal Ammoniac on March 02, 2022, 07:05:06 amWow, that's impressive. I just checked the one I use as a server, the last time it was rebooted was because I knocked the power cord out when I was moving things around. This one is a RPi4 and has been rock solid. I have it running all sorts of stuff and it has proven to be totally adequate.james@piserver:~ $ uptime10:59:53 up 344 days, 1:04, 1 user, load average: 0.00, 0.00, 0.00

author: ralphrmartin
author_rank: Frequent Contributor
timestamp: March 06, 2022, 09:27:05 pm
content: Quote from: Sal Ammoniac on March 02, 2022, 07:05:06 amSo it's bang up-to-date with security fixes, then???

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: March 06, 2022, 09:33:45 pm
content: Quote from: ralphrmartin on March 06, 2022, 09:27:05 pmPi and security don't belong in the same conversation.

author: rob77
author_rank: Super Contributor
timestamp: March 06, 2022, 10:08:48 pm
content: Quote from: BradC on February 27, 2022, 01:13:04 pmnot Linux... your distribution is doing than by dictating the default value of swapinessLinux (the kernel) is just doing what it's told to do. set your swapiness accordingly and the kernel will not page out anything unless it's the last resort option.https://www.howtogeek.com/449691/what-is-swapiness-on-linux-and-how-to-change-it/

author: DiTBho
author_rank: Super Contributor
timestamp: March 06, 2022, 11:35:03 pm
content: OK, for the top notorious-worst-ever list,MIPS64LE Loongson-3-5000: here the Linux kernel crashes in less than an hour. I think it's currently the less long-lived I have ever seen

author: james_s
author_rank: Super Contributor
timestamp: March 07, 2022, 04:19:33 am
content: Quote from: Ed.Kloonk on March 06, 2022, 09:33:45 pmWhy not? The RPi is just a computer, security is determined by the software and the way it's set up. It's a standard Linux distro, it's as secure as it is configured to be.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: March 07, 2022, 04:35:41 am
content: Quote from: james_s on March 07, 2022, 04:19:33 amShort answer. Too many black boxes.

author: BradC
author_rank: Super Contributor
timestamp: March 07, 2022, 05:39:17 am
content: Quote from: rob77 on March 06, 2022, 10:08:48 pmNothing to do with "my distribution" and entirely kernel defaults. So yes, Linux...There is nothing on any of my systems that tweak kernel policy in any way other than what I explicitly do and I don't touch the VM subsystem.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: March 07, 2022, 06:24:48 am
content: I wont argue but request/suggest users backup the SD card. The weeks and months of config files, scripts and exe's that you have accumulated since the install. I'm just asking if you were to put the SD card in your pocket, can you restore all your work?I'm all for solid state media decimation. Just back up your work. That is all.

author: rob77
author_rank: Super Contributor
timestamp: March 07, 2022, 09:31:56 am
content: Quote from: BradC on March 07, 2022, 05:39:17 am"your distribution" => linux distro you're using.. didn't say you personally changed anything...it's your linux distro maintainer who chose the default values at compile time. for example raspbian designed to run from a sd crad will have a low default swapiness but a desktop distro will have a higher one.

author: BradC
author_rank: Super Contributor
timestamp: March 07, 2022, 02:30:14 pm
content: Quote from: rob77 on March 07, 2022, 09:31:56 amI know what you're saying. I'm saying my "distribution" doesn't touch any knobs. My VM settings, and in fact everything except some block and md settings are absolutely kernel default and those I explicitly change.

author: rob77
author_rank: Super Contributor
timestamp: March 07, 2022, 03:41:59 pm
content: Quote from: BradC on March 07, 2022, 02:30:14 pmi see your point... but keeping the kernel source defaults as your distro's defaults still doesn't mean that "Linux will use swap to maximize buffers.." it's still the maintainer of the distro (be it you or anyone else) who decided that will be the behavior of that particular linux distro.it's like not choosing a color of your new car (even though it would be free of charge) and then stating that they sell only red ones because the one you've got is red.analogy to the above : you didn't pick a default at compile time and then stated that the behavior of your linux dristro is the behavior of Linux in general.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: March 11, 2022, 10:18:51 pm
content: Quote from: ralphrmartin on March 06, 2022, 09:27:05 pmIt's not connected to the Internet...

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: March 12, 2022, 12:51:28 am
content: Quote from: Sal Ammoniac on March 11, 2022, 10:18:51 pmAs far as we know..

author: cdev
author_rank: Super Contributor
timestamp: March 20, 2022, 07:32:49 pm
content: I have several Pis that run 24/7/365. As a distro, RaspiOS (ex Raspbian) seems very stable. Not quite as stable as freeBSD, but very good..If you are having crashes, look for a memory leak or some problem with one of the programs you are running.Check the logs.

author: e100
author_rank: Frequent Contributor
timestamp: April 04, 2022, 06:29:01 am
content: Just an update, this week I discovered that the 32bit version of Influxdb on the Raspberry Pi consumes huge amounts of CPU time if its database is allowed to get bigger than 2GB. Apparently this is a known problem that won't be fixed in the 32bit version as the Influx developers are focused on the 64bit version. (The Pi still isn't mainstream 64bit yet.)Strangely the default setting is for Influx databases to have no size limit, which of course is a ticking bomb. You'll either hit the 2GB limit or run out of disk space. The workaround is to set the 'retention policy' at database creation time.The CPU starvation had a particularly bad impact on my Java program as it throttled the rate at which it could send MQTT messages. It went from the desired 1 per second to 1 every 9 seconds. This in turn caused another program running on a connected microcontroller to think that there was an ongoing comms failure and therefore it went into auto-shutdown mode turning off the hardware it was controlling.Even stranger was that node-red running on the same Pi was affected to some degree, but not as badly affected as my Java program. No idea why that would be the case.Nothing crashed, all the program logic behaved as designed, there were no memory leaks, my Java program just got slower and slower until the keep-alive timeouts I had programmed into the system to detect hardware disconnections were triggered by software slowness. To fix it I deleted the database and recreated a new one with a 28 day retention policy and everything just went back to normal with MQTT messages being sent at the desired 1 second interval. I didn't need to restart any processes or reboot the Pi.Not sure how you would detect this kind of problem in a test environment except for leaving it running for months. Sure, if you had prior knowledge about a weakness then you could prime the system and wait for it to fail at a predicted time, but in this case who would of thought that there would be a sudden massive increase in CPU usage just because a database file has gone over some size limit. The whole point of databases is that they are good at storing stuff therefore you would assume that 2GB wouldn't be a problem.

author: cdev
author_rank: Super Contributor
timestamp: April 04, 2022, 04:37:30 pm
content: if you make 64 bit hw and os your defaults you'll see fewer problems. A lot fewer

author: brucehoult
author_rank: Super Contributor
timestamp: April 04, 2022, 09:57:23 pm
content: Quote from: e100 on April 04, 2022, 06:29:01 amShockingly.The first 64 bit Pis went on sale more than 6 years ago, and every Linux-capable Pi in the line-up is now 64 bit, even the Pi Zero 2.

author: SiliconWizard
author_rank: Super Contributor
timestamp: April 04, 2022, 10:03:41 pm
content: I've been using Archlinuxarm on the Pi, and there's a 64-bit version that works fairly well.https://archlinuxarm.org/They claim "near full support", I haven't personally run into issues with it, but possibly some hardware features might not have full support yet. Graphics acceleration works. But I don't use Pi's as any kind of "desktop" computing, so some things may be missing that do not bother me.As to the lag supporting 64-bit being shocking, yeah, but usual. As I routinely say, the real problem with all those SBCs, Pi included (but possibly to a lesser extent than some other brands), is software support.The hardware may be great, but software support is often lacking in some ways and you're just stuck with that. And since many of them use SoCs than do not have public documentation, you can't even help.

author: james_s
author_rank: Super Contributor
timestamp: April 05, 2022, 12:02:18 am
content: I think the reason for the lack of much 64 bit support has been that most of the RPis out there have too little memory to need 64 bit. None of mine have more than 4GB and for the things I use them for they don't need it. That's not to say there isn't a point in 64 bit or that in the future it won't be more common but so far it hasn't really been sorely needed.

author: SiliconWizard
author_rank: Super Contributor
timestamp: April 05, 2022, 12:12:00 am
content: That sure was one of the main points, but the Pi now has 8GB versions, so...

author: james_s
author_rank: Super Contributor
timestamp: April 05, 2022, 12:20:54 am
content: Quote from: SiliconWizard on April 05, 2022, 12:12:00 amIt does now, but that is a recent development, and the vast majority of those deployed have much less. 64 bit will mature over time, when the first version of the RPi came out things were pretty rough around the edges on that one too.

author: brucehoult
author_rank: Super Contributor
timestamp: April 05, 2022, 01:56:36 am
content: Quote from: james_s on April 05, 2022, 12:02:18 amOn both ARM and x86 there are a number of very good reasons to use the 64 bit ISA even if you don't need more than 4 GB of (virtual) memory  On both you get double the number of registers and a much more efficient function call ABI in the 64 bit ISA.In the RISC-V world there are 64 bit SoCs with as little as 8 MB RAM (Kendryte K210) or 64 MB running Linux (MangoPi-Nezha MQ with Allwinner D1s) and no one says "OMG they should only be 32 bit!". They don't even *have* a 32 bit mode -- 64 is all you get. and there are also reasons to use more than 4 GB of virtual memory even if you don't have that much physical memory -- sparce address spaces have a number of benefits, including security and fragmentation.

author: DiTBho
author_rank: Super Contributor
timestamp: April 05, 2022, 10:16:07 am
content: Quote from: brucehoult on April 05, 2022, 01:56:36 amLike MIPS4. 64bit is all you getp.s.I have here a R18K prototype board.R16K was the last commercial MIPS4.

author: brucehoult
author_rank: Super Contributor
timestamp: April 05, 2022, 01:34:27 pm
content: Quote from: DiTBho on April 05, 2022, 10:16:07 amIt's going to be interesting to see what MIPS does with RISC-V and whether it is successful.

author: cdev
author_rank: Super Contributor
timestamp: April 05, 2022, 06:46:07 pm
content: The OS yes, but the RPI3 only comes in a 1GB flavor which supoorts a 64 bit OS, there is no such thing as a 4GB RPI3. (unless you build it yourself, which might be possible, with the right RAM modules, if such exists, I suspect)

author: james_s
author_rank: Super Contributor
timestamp: April 05, 2022, 08:17:29 pm
content: The RAM on the RPI is integrated into the SOC and AFAIK you can't buy the CPU anywhere. Broadcom only sells them to large volume manufactures.

author: brucehoult
author_rank: Super Contributor
timestamp: April 06, 2022, 12:18:26 am
content: Quote from: cdev on April 05, 2022, 06:46:07 pmYou should be able to expect to run a 1 GB application (minus maybe 32 or 64 MB for the OS) on a machine wth 1 GB RAM. But in a 32 bit address space that is close enough to 4 GB (or 2 GB for user space) that all kinds of undesirable things start to happen:- a random pointer (or a malicious one) is likely to point to actual in-use data- a conservative garbage collector won't be able to tell that integers aren't poointers- if a malloc()-using application is repeatedly allocating and freeing objects of even a few dozen KB, let alone MB, then the address space can easily become so fragmented that it's not possible to allocate a large object even though there is plenty of space available.Running a 64 bit OS solves all these problems.The only downside is that pointers are twice as big in RAM, and also registers saved on the stack by function calls are twice as big. This can be a real problem for programs that use only a few KB of RAM, but programs that use 1 GB of RAM seldom use a significant percentage of that for stack or pointers.

author: DiTBho
author_rank: Super Contributor
timestamp: April 06, 2022, 11:25:35 am
content: the CPU module of my MIPS32board exports 32bit address, 32bit data, ctrl, DMA, IRQ, etc, but it reserves the highest 512Mbyte for the PCI and memory mapped stuffYou have { A31..A00 }, but you you cannot install 4GB of ram, the max you can practically address is 3.5GB

author: dave j
author_rank: Regular Contributor
timestamp: April 06, 2022, 12:14:02 pm
content: Quote from: james_s on April 05, 2022, 08:17:29 pmOnly on the Pi Zero 2 is the RAM integrated into the SOC. The BCM2835 based Pis (Pi1, CM1 and original Zeros) use PoP. All the other Pis have separate RAM chips soldered to the circuit board.

author: SiliconWizard
author_rank: Super Contributor
timestamp: April 06, 2022, 05:30:14 pm
content: Quote from: dave j on April 06, 2022, 12:14:02 pmYes. This was a new step with the SoC designed for the Zero 2. And it's limited to only one RAM size of course (making SoCs with multiple options for RAM size would be pretty expensive for no good reason.)Now I don't have any schematic for the Pi4 (for instance), so I don't know whether it would just be a matter of reworking the board (desoldering the existing RAM chip(s) and soldering new ones, assuming you have the equipement and skills to do it), or if the PCBs are different. Or if there is any other step to take, like solder jumpers, or something.

author: dave j
author_rank: Regular Contributor
timestamp: April 06, 2022, 06:30:43 pm
content: Quote from: SiliconWizard on April 06, 2022, 05:30:14 pmThere have been several queries about upgrading the RAM on the Raspberry Pi forums. They changed the circuit board for the Pi4 8GB to provide more power but I don't know if they are now using that board for all Pi4s now. There's also the issue of how would you get the SoC to recognise a different chip.

author: DiTBho
author_rank: Super Contributor
timestamp: April 06, 2022, 10:33:18 pm
content: Quote from: dave j on April 06, 2022, 06:30:43 pmIt looks like what we did in 2004 with Fonera, to upgrade the ram from 16 MB to 32 MB ... we had to - desolder and resolder - and - to hack the firmware - to make it recognize the different ram chip.It's always the same game