Help needed with some heap test code - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: April 29, 2023, 03:49:06 pm
content: This is just weird. I am using the Newlib arm32 ex Cube IDE heap code.My printf does not use the heap.Code:int blocksize=1000;while (true){osDelay(200);char *buf = malloc(blocksize);if (buf!=NULL){uint32_t addr = (uint32_t) buf;printf("malloc good, bk=%d addr=%08x",blocksize,addr);free(buf);blocksize+=200;}else{printf("malloc failed, bk=%d",blocksize);}}I am allocating a 1000 byte block, freeing it, and then repeating with a bigger one. It is supposed to fail around 57k bytes. The base of the heap area is shown correctly at 0x2000fb68. The top is 0x2001e000, which is 58520 bytes of heap space.The output is:malloc good, bk=1000 addr=2000fb68malloc good, bk=1200 addr=2000fb68malloc good, bk=1400 addr=2000fb68...malloc good, bk=29400 addr=2000fb68malloc good, bk=29600 addr=2000fb68malloc good, bk=29800 addr=2000fb68It fails at 29800+200 bytes. Way too soon.If I increment the blocksize by 1000 instead of 200, I get the same result, so it isn't caused by thenumber of timesmalloc/free get called.If I start with a blocksize of 57000, I getmalloc good, bk=57000 addr=2000fb68malloc good, bk=57200 addr=2000fb68malloc good, bk=57400 addr=2000fb68malloc good, bk=57600 addr=2000fb68malloc good, bk=57800 addr=2000fb68malloc good, bk=58000 addr=2000fb68malloc good, bk=58200 addr=2000fb68malloc good, bk=58400 addr=2000fb68which is correct!Working downwards from 57000 I find that the lowest initial blocksize which reaches the correct max blocksize is 55000. 54000 doesn't cut the mustard.This heap has been working for years, with TLS allocating and freeing 48k blocks, and of course (looking at the above) that works fine, millions of times (literally tested thus). But always 48k! I am working through a list of final things to test because I am writing up some user docs for this thing, and I had a hunch that this was never tested.Can someone please tell me whether I need to go watch some youtube videos on how to write C, or I need to replace the shitty heap codeSame result is obtained if there isnodebug output until failure, and incrementing withnodelays, in 1 byte blocksize increments.FWIW the malloc+free time is 10 microseconds, which is not bad considering there are four mutex calls there (FreeRTOS).====Digging around for alternative malloc code, and particularly looking around to see whether I have a bug in _sbrk, I came across thishttps://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.cwhere it statesQuoteThat is quite a huge limitation on heap usage, no?On _sbrk, I found e.g. thishttps://github.com/zephyrproject-rtos/zephyr/blob/main/lib/libc/newlib/libc-hooks.cwhich the original ST lib was based on, and which stupidly sets the heap limit at the current value of SP. This is dumb; I am setting it at the lowest stack address.Digging around some more, I found thishttps://stackoverflow.com/questions/39088598/malloc-in-newlib-does-it-waste-memory-after-one-big-failure-allocationwhich is not quite the above problem, but it shows that just because code is old doesn't mean it isn't buggy, and one has to be careful picking up sourcecode from around the net, especially when it is massively complicated as malloc is. Even the smallest lib is about 10k lines of C. Also it turns out that my ST lib newlib malloc doesn't have that problem10909: malloc failed, bk=6000011911: malloc failed, bk=5900012913: malloc good, bk=58000 addr=2000fb6812915: malloc good, bk=57000 addr=2000fb6812917: malloc good, bk=56000 addr=2000fb6812919: malloc good, bk=55000 addr=2000fb68

author: peter-h
author_rank: Super Contributor
timestamp: April 30, 2023, 07:21:12 am
content: A bit more data... with initial blocksize of 54000 it fails to use up all the RAM. With 55000 it works ok.With 54000, I get the following (showing sbrk input and output; numbers on the left are a milliseconds counter)10908: HEAP TEST10910: malloc sbrk, incr=54024, ret=2000fb6010912: malloc sbrk, incr=408, ret=2001ce6810914: malloc good, bk=54000 addr=2000fb6810926: malloc good, bk=54100 addr=2000fb6810938: malloc good, bk=54200 addr=2000fb6810950: malloc good, bk=54300 addr=2000fb6810962: malloc good, bk=54400 addr=2000fb6810974: malloc sbrk, incr=57344, ret=ffffffff10976: malloc failed, bk=54500With 55000 I get this10908: HEAP TEST10910: malloc sbrk, incr=55024, ret=2000fb6010912: malloc sbrk, incr=3504, ret=2001d25010914: malloc good, bk=55000 addr=2000fb6810926: malloc good, bk=55100 addr=2000fb6810938: malloc good, bk=55200 addr=2000fb6810950: malloc good, bk=55300 addr=2000fb6810962: malloc good, bk=55400 addr=2000fb6810974: malloc good, bk=55500 addr=2000fb6810986: malloc good, bk=55600 addr=2000fb6810998: malloc good, bk=55700 addr=2000fb6811010: malloc good, bk=55800 addr=2000fb6811022: malloc good, bk=55900 addr=2000fb6811034: malloc good, bk=56000 addr=2000fb6811046: malloc good, bk=56100 addr=2000fb6811058: malloc good, bk=56200 addr=2000fb6811070: malloc good, bk=56300 addr=2000fb6811082: malloc good, bk=56400 addr=2000fb6811094: malloc good, bk=56500 addr=2000fb6811106: malloc good, bk=56600 addr=2000fb6811118: malloc good, bk=56700 addr=2000fb6811130: malloc good, bk=56800 addr=2000fb6811142: malloc good, bk=56900 addr=2000fb6811154: malloc good, bk=57000 addr=2000fb6811166: malloc good, bk=57100 addr=2000fb6811178: malloc good, bk=57200 addr=2000fb6811190: malloc good, bk=57300 addr=2000fb6811202: malloc good, bk=57400 addr=2000fb6811214: malloc good, bk=57500 addr=2000fb6811226: malloc good, bk=57600 addr=2000fb6811238: malloc good, bk=57700 addr=2000fb6811250: malloc good, bk=57800 addr=2000fb6811262: malloc good, bk=57900 addr=2000fb6811274: malloc good, bk=58000 addr=2000fb6811286: malloc good, bk=58100 addr=2000fb6811298: malloc good, bk=58200 addr=2000fb6811310: malloc good, bk=58300 addr=2000fb6811322: malloc good, bk=58400 addr=2000fb6811334: malloc good, bk=58500 addr=2000fb6811346: malloc sbrk, incr=61440, ret=ffffffff11348: malloc failed, bk=58600I reckon it is obvious, but can't get my head around itThis is _sbrkCode:// This is used by malloc().// The original Newlib version of this, on which the ST code was based// https://github.com/zephyrproject-rtos/zephyr/blob/main/lib/libc/newlib/libc-hooks.c// allowed the heap to go all the way up to the current SP value, which is stupid.// This one sets the limit at the base (lowest memory address) of the stack area.caddr_t _sbrk(int incr){// These two are defined in the linkfileextern char end asm("_end");// end of BSSextern char top asm("_top");// base of the general stackstatic char *heap_end;// this gets initialised to NULL by C conventionchar *prev_heap_end;// this gets initialised on 1st call here// This sets heap_end to end of BSS, on the first call to _sbrkif (heap_end == NULL)heap_end = &end;prev_heap_end = heap_end;// top = top of RAM minus size of stackif ( (heap_end + incr) > &top ){errno = ENOMEM;// not apparently used by anythingprev_heap_end = -1;}else{heap_end += incr;}printf("malloc sbrk, incr=%d, ret=%08x",(int)incr, (int) prev_heap_end); // TODOreturn (caddr_t) prev_heap_end;}If I start with a blocksize of 1000, we get a lot more output... sorry about thatCode:10908: HEAP TEST10910: malloc sbrk, incr=1024, ret=2000fb6010912: malloc sbrk, incr=160, ret=2000ff6010914: malloc good, bk=1000 addr=2000fb6810926: malloc good, bk=1100 addr=2000fb6810938: malloc sbrk, incr=4096, ret=2001000010940: malloc good, bk=1200 addr=2000fb6810952: malloc good, bk=1300 addr=2000fb6810964: malloc good, bk=1400 addr=2000fb6810976: malloc good, bk=1500 addr=2000fb6810988: malloc good, bk=1600 addr=2000fb6811000: malloc good, bk=1700 addr=2000fb6811012: malloc good, bk=1800 addr=2000fb6811024: malloc good, bk=1900 addr=2000fb6811036: malloc good, bk=2000 addr=2000fb6811048: malloc good, bk=2100 addr=2000fb6811060: malloc good, bk=2200 addr=2000fb6811072: malloc good, bk=2300 addr=2000fb6811084: malloc good, bk=2400 addr=2000fb6811096: malloc good, bk=2500 addr=2000fb6811108: malloc good, bk=2600 addr=2000fb6811120: malloc good, bk=2700 addr=2000fb6811132: malloc good, bk=2800 addr=2000fb6811144: malloc good, bk=2900 addr=2000fb6811156: malloc good, bk=3000 addr=2000fb6811168: malloc good, bk=3100 addr=2000fb6811180: malloc good, bk=3200 addr=2000fb6811192: malloc good, bk=3300 addr=2000fb6811204: malloc good, bk=3400 addr=2000fb6811216: malloc good, bk=3500 addr=2000fb6811228: malloc good, bk=3600 addr=2000fb6811240: malloc good, bk=3700 addr=2000fb6811252: malloc good, bk=3800 addr=2000fb6811264: malloc good, bk=3900 addr=2000fb6811276: malloc good, bk=4000 addr=2000fb6811288: malloc good, bk=4100 addr=2000fb6811300: malloc good, bk=4200 addr=2000fb6811312: malloc good, bk=4300 addr=2000fb6811324: malloc good, bk=4400 addr=2000fb6811336: malloc good, bk=4500 addr=2000fb6811348: malloc good, bk=4600 addr=2000fb6811360: malloc good, bk=4700 addr=2000fb6811372: malloc good, bk=4800 addr=2000fb6811384: malloc good, bk=4900 addr=2000fb6811396: malloc good, bk=5000 addr=2000fb6811408: malloc good, bk=5100 addr=2000fb6811420: malloc good, bk=5200 addr=2000fb6811432: malloc sbrk, incr=8192, ret=2001100011434: malloc good, bk=5300 addr=2000fb6811446: malloc good, bk=5400 addr=2000fb6811458: malloc good, bk=5500 addr=2000fb6811470: malloc good, bk=5600 addr=2000fb6811482: malloc good, bk=5700 addr=2000fb6811494: malloc good, bk=5800 addr=2000fb6811506: malloc good, bk=5900 addr=2000fb6811518: malloc good, bk=6000 addr=2000fb6811530: malloc good, bk=6100 addr=2000fb6811542: malloc good, bk=6200 addr=2000fb6811554: malloc good, bk=6300 addr=2000fb6811566: malloc good, bk=6400 addr=2000fb6811578: malloc good, bk=6500 addr=2000fb6811590: malloc good, bk=6600 addr=2000fb6811602: malloc good, bk=6700 addr=2000fb6811614: malloc good, bk=6800 addr=2000fb6811626: malloc good, bk=6900 addr=2000fb6811638: malloc good, bk=7000 addr=2000fb6811650: malloc good, bk=7100 addr=2000fb6811662: malloc good, bk=7200 addr=2000fb6811674: malloc good, bk=7300 addr=2000fb6811686: malloc good, bk=7400 addr=2000fb6811698: malloc good, bk=7500 addr=2000fb6811710: malloc good, bk=7600 addr=2000fb6811722: malloc good, bk=7700 addr=2000fb6811734: malloc good, bk=7800 addr=2000fb6811746: malloc good, bk=7900 addr=2000fb6811758: malloc good, bk=8000 addr=2000fb6811770: malloc good, bk=8100 addr=2000fb6811782: malloc good, bk=8200 addr=2000fb6811794: malloc good, bk=8300 addr=2000fb6811806: malloc good, bk=8400 addr=2000fb6811818: malloc good, bk=8500 addr=2000fb6811830: malloc good, bk=8600 addr=2000fb6811842: malloc good, bk=8700 addr=2000fb6811854: malloc good, bk=8800 addr=2000fb6811866: malloc good, bk=8900 addr=2000fb6811878: malloc good, bk=9000 addr=2000fb6811890: malloc good, bk=9100 addr=2000fb6811902: malloc good, bk=9200 addr=2000fb6811914: malloc good, bk=9300 addr=2000fb6811926: malloc good, bk=9400 addr=2000fb6811938: malloc good, bk=9500 addr=2000fb6811950: malloc good, bk=9600 addr=2000fb6811962: malloc good, bk=9700 addr=2000fb6811974: malloc good, bk=9800 addr=2000fb6811986: malloc good, bk=9900 addr=2000fb6811998: malloc good, bk=10000 addr=2000fb6812010: malloc good, bk=10100 addr=2000fb6812022: malloc good, bk=10200 addr=2000fb6812034: malloc good, bk=10300 addr=2000fb6812046: malloc good, bk=10400 addr=2000fb6812058: malloc good, bk=10500 addr=2000fb6812070: malloc good, bk=10600 addr=2000fb6812082: malloc good, bk=10700 addr=2000fb6812094: malloc good, bk=10800 addr=2000fb6812106: malloc good, bk=10900 addr=2000fb6812118: malloc good, bk=11000 addr=2000fb6812130: malloc good, bk=11100 addr=2000fb6812142: malloc good, bk=11200 addr=2000fb6812154: malloc good, bk=11300 addr=2000fb6812166: malloc good, bk=11400 addr=2000fb6812178: malloc good, bk=11500 addr=2000fb6812190: malloc good, bk=11600 addr=2000fb6812202: malloc good, bk=11700 addr=2000fb6812214: malloc good, bk=11800 addr=2000fb6812226: malloc good, bk=11900 addr=2000fb6812238: malloc good, bk=12000 addr=2000fb6812250: malloc good, bk=12100 addr=2000fb6812262: malloc good, bk=12200 addr=2000fb6812274: malloc good, bk=12300 addr=2000fb6812286: malloc good, bk=12400 addr=2000fb6812298: malloc good, bk=12500 addr=2000fb6812310: malloc good, bk=12600 addr=2000fb6812322: malloc good, bk=12700 addr=2000fb6812334: malloc good, bk=12800 addr=2000fb6812346: malloc good, bk=12900 addr=2000fb6812358: malloc good, bk=13000 addr=2000fb6812370: malloc good, bk=13100 addr=2000fb6812382: malloc good, bk=13200 addr=2000fb6812394: malloc good, bk=13300 addr=2000fb6812406: malloc good, bk=13400 addr=2000fb6812418: malloc sbrk, incr=16384, ret=2001300012420: malloc good, bk=13500 addr=2000fb6812432: malloc good, bk=13600 addr=2000fb6812444: malloc good, bk=13700 addr=2000fb6812456: malloc good, bk=13800 addr=2000fb6812468: malloc good, bk=13900 addr=2000fb6812480: malloc good, bk=14000 addr=2000fb6812492: malloc good, bk=14100 addr=2000fb6812504: malloc good, bk=14200 addr=2000fb6812516: malloc good, bk=14300 addr=2000fb6812528: malloc good, bk=14400 addr=2000fb6812540: malloc good, bk=14500 addr=2000fb6812552: malloc good, bk=14600 addr=2000fb6812564: malloc good, bk=14700 addr=2000fb6812576: malloc good, bk=14800 addr=2000fb6812588: malloc good, bk=14900 addr=2000fb6812600: malloc good, bk=15000 addr=2000fb6812612: malloc good, bk=15100 addr=2000fb6812624: malloc good, bk=15200 addr=2000fb6812636: malloc good, bk=15300 addr=2000fb6812648: malloc good, bk=15400 addr=2000fb6812660: malloc good, bk=15500 addr=2000fb6812672: malloc good, bk=15600 addr=2000fb6812684: malloc good, bk=15700 addr=2000fb6812696: malloc good, bk=15800 addr=2000fb6812708: malloc good, bk=15900 addr=2000fb6812720: malloc good, bk=16000 addr=2000fb6812732: malloc good, bk=16100 addr=2000fb6812744: malloc good, bk=16200 addr=2000fb6812756: malloc good, bk=16300 addr=2000fb6812768: malloc good, bk=16400 addr=2000fb6812780: malloc good, bk=16500 addr=2000fb6812792: malloc good, bk=16600 addr=2000fb6812804: malloc good, bk=16700 addr=2000fb6812816: malloc good, bk=16800 addr=2000fb6812828: malloc good, bk=16900 addr=2000fb6812840: malloc good, bk=17000 addr=2000fb6812852: malloc good, bk=17100 addr=2000fb6812864: malloc good, bk=17200 addr=2000fb6812876: malloc good, bk=17300 addr=2000fb6812888: malloc good, bk=17400 addr=2000fb6812900: malloc good, bk=17500 addr=2000fb6812912: malloc good, bk=17600 addr=2000fb6812924: malloc good, bk=17700 addr=2000fb6812936: malloc good, bk=17800 addr=2000fb6812948: malloc good, bk=17900 addr=2000fb6812960: malloc good, bk=18000 addr=2000fb6812972: malloc good, bk=18100 addr=2000fb6812984: malloc good, bk=18200 addr=2000fb6812996: malloc good, bk=18300 addr=2000fb6813008: malloc good, bk=18400 addr=2000fb6813020: malloc good, bk=18500 addr=2000fb6813032: malloc good, bk=18600 addr=2000fb6813044: malloc good, bk=18700 addr=2000fb6813056: malloc good, bk=18800 addr=2000fb6813068: malloc good, bk=18900 addr=2000fb6813080: malloc good, bk=19000 addr=2000fb6813092: malloc good, bk=19100 addr=2000fb6813104: malloc good, bk=19200 addr=2000fb6813116: malloc good, bk=19300 addr=2000fb6813128: malloc good, bk=19400 addr=2000fb6813140: malloc good, bk=19500 addr=2000fb6813152: malloc good, bk=19600 addr=2000fb6813164: malloc good, bk=19700 addr=2000fb6813176: malloc good, bk=19800 addr=2000fb6813188: malloc good, bk=19900 addr=2000fb6813200: malloc good, bk=20000 addr=2000fb6813212: malloc good, bk=20100 addr=2000fb6813224: malloc good, bk=20200 addr=2000fb6813236: malloc good, bk=20300 addr=2000fb6813248: malloc good, bk=20400 addr=2000fb6813260: malloc good, bk=20500 addr=2000fb6813272: malloc good, bk=20600 addr=2000fb6813284: malloc good, bk=20700 addr=2000fb6813296: malloc good, bk=20800 addr=2000fb6813308: malloc good, bk=20900 addr=2000fb6813320: malloc good, bk=21000 addr=2000fb6813332: malloc good, bk=21100 addr=2000fb6813344: malloc good, bk=21200 addr=2000fb6813356: malloc good, bk=21300 addr=2000fb6813368: malloc good, bk=21400 addr=2000fb6813380: malloc good, bk=21500 addr=2000fb6813392: malloc good, bk=21600 addr=2000fb6813404: malloc good, bk=21700 addr=2000fb6813416: malloc good, bk=21800 addr=2000fb6813428: malloc good, bk=21900 addr=2000fb6813440: malloc good, bk=22000 addr=2000fb6813452: malloc good, bk=22100 addr=2000fb6813464: malloc good, bk=22200 addr=2000fb6813476: malloc good, bk=22300 addr=2000fb6813488: malloc good, bk=22400 addr=2000fb6813500: malloc good, bk=22500 addr=2000fb6813512: malloc good, bk=22600 addr=2000fb6813524: malloc good, bk=22700 addr=2000fb6813536: malloc good, bk=22800 addr=2000fb6813548: malloc good, bk=22900 addr=2000fb6813560: malloc good, bk=23000 addr=2000fb6813572: malloc good, bk=23100 addr=2000fb6813584: malloc good, bk=23200 addr=2000fb6813596: malloc good, bk=23300 addr=2000fb6813608: malloc good, bk=23400 addr=2000fb6813620: malloc good, bk=23500 addr=2000fb6813632: malloc good, bk=23600 addr=2000fb6813644: malloc good, bk=23700 addr=2000fb6813656: malloc good, bk=23800 addr=2000fb6813668: malloc good, bk=23900 addr=2000fb6813680: malloc good, bk=24000 addr=2000fb6813692: malloc good, bk=24100 addr=2000fb6813704: malloc good, bk=24200 addr=2000fb6813716: malloc good, bk=24300 addr=2000fb6813728: malloc good, bk=24400 addr=2000fb6813740: malloc good, bk=24500 addr=2000fb6813752: malloc good, bk=24600 addr=2000fb6813764: malloc good, bk=24700 addr=2000fb6813776: malloc good, bk=24800 addr=2000fb6813788: malloc good, bk=24900 addr=2000fb6813800: malloc good, bk=25000 addr=2000fb6813812: malloc good, bk=25100 addr=2000fb6813824: malloc good, bk=25200 addr=2000fb6813836: malloc good, bk=25300 addr=2000fb6813848: malloc good, bk=25400 addr=2000fb6813860: malloc good, bk=25500 addr=2000fb6813872: malloc good, bk=25600 addr=2000fb6813884: malloc good, bk=25700 addr=2000fb6813896: malloc good, bk=25800 addr=2000fb6813908: malloc good, bk=25900 addr=2000fb6813920: malloc good, bk=26000 addr=2000fb6813932: malloc good, bk=26100 addr=2000fb6813944: malloc good, bk=26200 addr=2000fb6813956: malloc good, bk=26300 addr=2000fb6813968: malloc good, bk=26400 addr=2000fb6813980: malloc good, bk=26500 addr=2000fb6813992: malloc good, bk=26600 addr=2000fb6814004: malloc good, bk=26700 addr=2000fb6814016: malloc good, bk=26800 addr=2000fb6814028: malloc good, bk=26900 addr=2000fb6814040: malloc good, bk=27000 addr=2000fb6814052: malloc good, bk=27100 addr=2000fb6814064: malloc good, bk=27200 addr=2000fb6814076: malloc good, bk=27300 addr=2000fb6814088: malloc good, bk=27400 addr=2000fb6814100: malloc good, bk=27500 addr=2000fb6814112: malloc good, bk=27600 addr=2000fb6814124: malloc good, bk=27700 addr=2000fb6814136: malloc good, bk=27800 addr=2000fb6814148: malloc good, bk=27900 addr=2000fb6814160: malloc good, bk=28000 addr=2000fb6814172: malloc good, bk=28100 addr=2000fb6814184: malloc good, bk=28200 addr=2000fb6814196: malloc good, bk=28300 addr=2000fb6814208: malloc good, bk=28400 addr=2000fb6814220: malloc good, bk=28500 addr=2000fb6814232: malloc good, bk=28600 addr=2000fb6814244: malloc good, bk=28700 addr=2000fb6814256: malloc good, bk=28800 addr=2000fb6814268: malloc good, bk=28900 addr=2000fb6814280: malloc good, bk=29000 addr=2000fb6814292: malloc good, bk=29100 addr=2000fb6814304: malloc good, bk=29200 addr=2000fb6814316: malloc good, bk=29300 addr=2000fb6814328: malloc good, bk=29400 addr=2000fb6814340: malloc good, bk=29500 addr=2000fb6814352: malloc good, bk=29600 addr=2000fb6814364: malloc good, bk=29700 addr=2000fb6814376: malloc good, bk=29800 addr=2000fb6814388: malloc sbrk, incr=32768, ret=ffffffff14390: malloc failed, bk=29900

author: ejeffrey
author_rank: Super Contributor
timestamp: April 30, 2023, 03:29:39 pm
content: One possibility is that malloc() or free() themselves are occasionally allocating memory to manage the free list, and if that allocation lands in the middle of your memory segment it then limits the maximum size allocation that can succeed.

author: peter-h
author_rank: Super Contributor
timestamp: April 30, 2023, 04:54:50 pm
content: Yeah... but that would be a really serious bug, no?I wonder if thishttps://stackoverflow.com/questions/39088598/malloc-in-newlib-does-it-waste-memory-after-one-big-failure-allocation/76138157#76138157is related. It refers to mallocr.c. The problem described there I don't have (I tested it). I made a post there but some mod deleted it so I don't want to waste my time on that weird site where most replies are totally off the mark anyway.In my product, the heap is used to allocate some buffers at startup (according to product options in a config file) and these are never freed. Then there is TLS allocating 48k and freeing it when the session ends, and this repeats many times (and works ok, 48k every time).What makes it fail is allocatingincreasingsize blocks.

author: DavidAlfa
author_rank: Super Contributor
timestamp: April 30, 2023, 05:23:28 pm
content: Yes, for some reason it discards the previously allocated block if the new request is larger.Thus, first allocating 1k+1k, freeing them and then allocating a 2kb block won't use the area previously used by 1k+1k.But if you allocate 2KB, then it will be used by 1k+1k.So my workaround (example) was to allocate the entire RAM at boot, then freeing it.Now malloc is aware of the entire pool and behaves as expected.Of course, fragmentation will be a different issue.

author: Siwastaja
author_rank: Super Contributor
timestamp: April 30, 2023, 05:32:08 pm
content: The whole point of malloc() is it stores metadata for free().If all you need is one-time allocation of things depending on configuration (which is only read at boot time) and no free, writing your own "replacement" for malloc would be an order of magnitude simpler than doing your own malloc from scratch. In other words, you would have your own reliable and less wasteful allocation system working faster than it takes to debug the Newlib's malloc.Something with this idea:Code:uint8_t my_buf;void* cur_ptr = my_buf;if(config.thing_a_enabled){// equivalent of malloc(555):thing_a_buf = cur_ptr;cur_ptr += 555;}if(config.thing_b_enabled)....Add alignment and range check and that's pretty much it, no?And if you have a few combinations only, then it's even simpler to just use a union.

author: peter-h
author_rank: Super Contributor
timestamp: April 30, 2023, 06:44:51 pm
content: Quote*I* am always freeing each block immediately afterwards.The interesting observation is that each newly allocated block is being allocated at the base of the heap area. I've read that a lot of heap code doesn't do that. So that is a plus, which should prevent the behaviour described here.QuoteSee above; that is true but I also have that 48k TLS block. It just happens to be the same size every time, which is why it worksQuoteIt works!!! This is the code I used; same as yours basically, and it will go in main.c before any heap usageCode:/** This establishes the biggest block which can be allocated on the heap, allocates* it and frees it. This is necessary to work around a bug in ST's Newlib heap code* which fails to use the whole heap area if some smaller blocks were allocated* previously.* One has to start with the max heap space (&top-&end) and loop down from there* because we don't know exactly what initial allocation overhead there is in the* heap code. Actually it is only about 5x8 bytes.* There is a limit counter to prevent some huge overrun if top<end, etc.**/extern char end asm("_end");// end of BSS (from linker script)extern char top asm("_top");// base of the general stack (as above)char *heap_ptr = NULL;uint32_t test_block = &top-&end;// max theoretical heap areauint32_t test_limit = 1000;// counter to set an upper limit on loopingwhile((heap_ptr==NULL) && (test_limit>0)){heap_ptr = malloc(test_block);test_block-=8;test_limit--;}free(heap_ptr);It's funny that you came across this bug tooQuoteI also notice that _sbrk is not getting called until the very end when it fails i.e. I am no longer seeing the various calls with 4096, 8192, etc. The first five sbrk calls below are from the above loop. The rest is as previously10908: HEAP TEST10910: malloc sbrk, incr=58552, ret=ffffffff10912: malloc sbrk, incr=58544, ret=ffffffff10914: malloc sbrk, incr=58536, ret=ffffffff10916: malloc sbrk, incr=58528, ret=2000fb6010918: malloc sbrk, incr=0, ret=2001e00010920: malloc good, bk=1000 addr=2000fb6810932: malloc good, bk=1100 addr=2000fb6810944: malloc good, bk=1200 addr=2000fb6810956: malloc good, bk=1300 addr=2000fb6810968: malloc good, bk=1400 addr=2000fb6810980: malloc good, bk=1500 addr=2000fb68...17808: malloc good, bk=58400 addr=2000fb6817820: malloc good, bk=58500 addr=2000fb6817832: malloc sbrk, incr=61440, ret=ffffffff17834: malloc failed, bk=58600

author: ejeffrey
author_rank: Super Contributor
timestamp: April 30, 2023, 07:23:01 pm
content: Quote from: peter-h on April 30, 2023, 04:54:50 pmDepends on what you mean by bug. As long as it returns 0 when unable to fulfill your request it is not doing anything incorrect. Doesn't mean it works for every situation.QuoteIf the question was correctly answered, including a reference to a bug fix submitted to the upstream package, and you posted a comment about a different bug, of course it got deleted. That is exactly working as intended. It is part of how SO keeps their answers mostly focused, and as a result is 100x more useful as a search result than discussion forms like eevblog,QuoteIt definitely sounds like non-ideal behavior, and possibly a bug in newlib. However, heap fragmentation is an unsolvable problem in this environment unless you are willing to make every allocation relocatable by using handles (which carries it's own costs and limitations). There is always a allocation pattern that is going to leave most of the memory free but unable to handle even a medium sized request. malloc() and free() are normally optimized around the case of allocating objects much smaller than available memory, and doing so with relatively low memory and CPU overhead. It's why a lot of people say "don't use dynamic memory allocation in microcontrollers" which is too general to be correct, but is understandable.

author: peter-h
author_rank: Super Contributor
timestamp: April 30, 2023, 07:48:29 pm
content: The problem is that if say you design your RAM to have 60k heap space, and malloc returns a 0 for a 29k block, that is not rightIn this case I have worked hard all around to make as much RAM as possible so that TLS is able to malloc its 48k. TLS doesn't really work with much less than that - unless you controlbothends and then you can do what you like.Fragmentation I do understand. It's an entirely different problem.I too don't use the heap in embedded, except as a one-off at startup as mentioned previously, but in the case of TLS there isn't much of a choice. It is- an optional feature, so a startup malloc makes sense- it can be invoked by different RTOS tasks (it is mutexed)- not enabled as such in a config file; it gets started up when an RTOS task invokes itIf it wasn't for the last one, and its use was enabled by a config file parameter, TLS could just use one malloc which is never freed. Yes this is debatable both ways. But fragmentation should be impossible if- all blocks except TLS are allocated only at startup (and these are typically only a few k)- TLS allocates and frees a 48k block (repeatedly)- total heap space is 57kIf one had several functions like TLS and they were allocating say 10k each, and there was no mutual exclusion on these executing, then fragmentation would be very possible.This has been another 3 day rabbit hole for meCan C even do the "garbage collection" thing? AFAICT that needs double indirection on all blocks, so the source code doesn't need to care of the address of the block has been physicaly changed.Re SO, that would be off topicbut I often see a question to which nearly all the answers are simply BS.

author: peter-h
author_rank: Super Contributor
timestamp: May 02, 2023, 02:36:48 pm
content: This gets even more funny.I reckon lots of people knew about this but nobody, apart fromDavidAlfa, looked into it properly.Looking at a 2019 (very early, before I got involved) version of my project, it has code in main.c to malloc and free a 48k block, and it talks about a bug in the heap code:Code:// Allocate and immediately free the entire heap memory in order that it becomes available to RTOS threads// This is a workaround for an RTOS heap management bugvoid *dummy = malloc(48*1024);free(dummy);But that malloc() would have gone to thenewlibheap code. Nothing to do with the RTOS heap (heap_4.c) whereby you give FreeRTOS a chuck of RAM and it runs its own private heap within that.Currently I am using the 64k CCM for the RTOS RAM block (which is at least partly a "heap").Just like LWIP does, in the infamous and crappily documented#define MEM_SIZE (6*1024)and just like TLS does (currently it is given a 48k block).Otherwise I need to test the RTOS heap also (and the TLS heap, and the LWIP heap).This stuff could all be buggy, if they use any of a number of heap sourcecode found online.I have not yet found a definitive sourcecode for the newlib heap I am using. Based on Cube IDE disassembly, it should contain calls to empty functions for obtaining and returning mutexes. I found sourcecode for the newlib printf family.More tests... I think this heap doesn't do what I think is called compaction. If you malloc/free and repeat, it does each block at the same address - the base of the heap. Buf if you are running two blocks, say one 5k and one 48k, the 5k one ends up quite high up (when it got allocated after the 48k one) and then when the 48k one stops being allocated, the 5k block never drops back down. I don't think this is "fragmentation"...

author: DiTBho
author_rank: Super Contributor
timestamp: May 02, 2023, 02:50:16 pm
content: Code:boolean_t is_bug;is_bug = (measured isNotEqualTo expected);which brings to-1- are you sure you have planned correctly what you expect?-2- are you sure you measure it correctly?

author: dare
author_rank: Contributor
timestamp: May 03, 2023, 05:01:04 pm
content: Quote from: peter-h on May 02, 2023, 02:36:48 pmGenerally speaking, malloc comes with the toolchain, as part of libc. If you're using a relatively modern toolchain (and building with --specs=nano.specs) this is the version of libc malloc you should be using:https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/nano-mallocr.c;h=a2b50facc35ebdfb4fc3c8fb11cd170d634649b3;hb=HEAD. (Note that there are #defines that map to the nano_* names onto regular malloc/free names).One quick way to tell if you're using this malloc is look for the following symbols in your linker map file: __malloc_free_list, __malloc_sbrk_start, sbrk_alignedIf you're not using this malloc, then I reckon you should switch toolchains. Also, I can't quite tell from your previous comments whether you've setup newlib's retargetable locking mechanism to work with FreeRTOS, but if not you will need to do that.FWIW, I ran the following code in a test application I'm building and got perfectly acceptable behavior for a total heap size of 260012 bytes:Code:void TestHeap(void){int blocksize = 1000;int increment = 1023;Log("Total Heap Size: %" PRId32 "\n", (uint32_t)GetHeapTotalSize());while (true){vTaskDelay(pdMS_TO_TICKS(200));char *buf = malloc(blocksize);if (buf!=NULL){Log("malloc good, bk=%d addr=0x%08" PRIX32 "\n", blocksize, (uint32_t)buf);free(buf);blocksize += increment;}else{Log("malloc failed, bk=%d\n",blocksize);blocksize -= increment;increment /= 2;if (increment == 0)break;blocksize += increment;}}}Code:Total Heap Size: 260012malloc good, bk=1000 addr=0x20010858malloc good, bk=2023 addr=0x20010C48malloc good, bk=3046 addr=0x20011438malloc good, bk=4069 addr=0x20010858...malloc good, bk=256750 addr=0x20010858malloc good, bk=257773 addr=0x20010858malloc good, bk=258796 addr=0x20010858malloc good, bk=259819 addr=0x20010858malloc failed, bk=260842malloc failed, bk=260330malloc failed, bk=260074malloc good, bk=259946 addr=0x20010858malloc failed, bk=260073malloc failed, bk=260009malloc good, bk=259977 addr=0x20010858malloc failed, bk=260008malloc good, bk=259992 addr=0x20010858malloc failed, bk=260007malloc good, bk=259999 addr=0x20010858malloc failed, bk=260006malloc failed, bk=260002malloc good, bk=260000 addr=0x20010858malloc failed, bk=260001

author: peter-h
author_rank: Super Contributor
timestamp: May 04, 2023, 11:36:33 am
content: Nope, those symbols are not found.I don't think I am using newlib-nano. I can't remember the decision chain now but it may have been to do with support for double floats, which I do want.This is my current setting:But - see below - those parts of it which reside in libc.a have been bypassed.QuoteI edited the heap code in (the weakened) libc.a to protect calls to malloc and freeCode:/** newlib_locking.c** Created on: 24 Jul 2022* Author: peter** These functions replace the empty ones in the newlib heap code.* See notes under LIBC on what this relates to - readme.txt.* We don't have the source for the libc.a library so this code replaces* the empty mutex stubs.** Based around* https://gist.github.com/thomask77/3a2d54a482c294beec5d87730e163bdd** Also see* https://www.eevblog.com/forum/programming/st-cube-gcc-how-to-override-a-function-not-defined-as-weak-(no-sources)** 27/7/22PHCurrently implements just heap mutexes.****/#include "FreeRTOS.h"#include "cmsis_os2.h"#include <newlib_locking.h>extern osMutexId_t g_HEAP_Mutex;// The mutex lock is *not* recursive (as is shown in many examples online) because// that seems pointless.// These two functions are used by both malloc and free.void __malloc_lock(void){osMutexAcquire(g_HEAP_Mutex,osWaitForever);}void __malloc_unlock (void){osMutexRelease(g_HEAP_Mutex);}// This one is not usedvoid __malloc_lock_acquire(void){osMutexAcquire(g_HEAP_Mutex,osWaitForever);}The heap code in libc.a contained called to __malloc_lock and __malloc_unlock. Once libc.a was weakened (I could have done just the required functions but it was just as easy to do the whole lib) the above functions replaced the dummy stubs in libc.a.Various people online have since commented that this is all BS and that there are much smarter ways but I never found any. And assertions that libc.aisweak are certainly BS for the code which came with Cube IDE c. 2019 which is what I am working with.Exhaustive multi-thread tests over months suggest this is working as intended.I then froze a local copy of libc.a so future Cube (and occassionally GCC) updated do not affect it. I spent some time identifying which of 66 (just counted them) libc.a files is the right one, made a local copy, and did a batch script to weaken it and replace the offending functions. There are posts here about ithttps://www.eevblog.com/forum/microcontrollers/is-st-cube-ide-a-piece-of-buggy-crap/msg4363183/#msg4363183The most important criteria were that it does hardware floats and supports the 32F4; most of the 66 files are obviously not applicable but some are close.