Open large files on 32-bit OS and ANSI C - Page 1
Forum section: electronics / general

author: Karel
author_rank: Super Contributor
timestamp: August 27, 2023, 12:22:35 pm
content: Using ANSI C, is there already a way to read/write large files (>2.14.2GB) on a 32-bit OS?

author: ledtester
author_rank: Super Contributor
timestamp: August 27, 2023, 01:45:40 pm
content: It will depend on the OS.For Linux/Unix I found this:Portable support for large fileshttps://stackoverflow.com/a/30867832which points to calls such as fseeko(), ftello(), fseeko64(), ftello64() and lseek64().For WinXP SP2 you have the Set/GetFilePointerEx() pair of calls.

author: Karel
author_rank: Super Contributor
timestamp: August 27, 2023, 02:44:05 pm
content: Quote from: ledtester on August 27, 2023, 01:45:40 pmIt's notANSI C

author: ejeffrey
author_rank: Super Contributor
timestamp: August 27, 2023, 03:13:37 pm
content: Yes. You can fopen() a file of any size and read it continuously until EOF.What you can't do is fseek() in a single step. Although technically you can call fseek(fh, LONG_MAX, SEEK_CUR) in a loop to advance to an arbitrary position.

author: Karel
author_rank: Super Contributor
timestamp: August 27, 2023, 03:35:31 pm
content: Quote from: ejeffrey on August 27, 2023, 03:13:37 pmInteresting. But the limitation of fseek(0 and ftell() seems cumbersome.It's weird that ANSI C has never been updated with 64-bit variants (that work on 32-bit hosts).

author: TheCalligrapher
author_rank: Regular Contributor
timestamp: August 27, 2023, 04:10:18 pm
content: Quote from: Karel on August 27, 2023, 12:22:35 pmThe question, as stated, makes no sense. There's no connection of any kind between bitness of OS and ANSI C. ANSI C does not dictate any limitation on the file size, tied to either the bitness of OS or anything else.The answer depends purely on implementation. If an implementation decides to support reading terabyte-sized files under a 8-bit OS - it can do so (under assumption that such files actually exist). It will require some extra effort, but there's nothing difficult about it. Or it can decide not to support such files.Note that the oft-mentioned problem with file positioning functions (e.g. `ftell`/`fseek`) and large files already exists even in 64-bit-specific environments. Due to poor design of C standard library, these functions expose naked `long` as the type that specifies absolute file position. This is a signed type, meaning that 64-bit-wide `long` is already insufficient to cover all possible absolute positions and sizes available in unsigned 64-bit-wide range. In other words, in an environment with even narrower `long` the problem is more pronounced in purely quantitative terms, but it is not really new.

author: ejeffrey
author_rank: Super Contributor
timestamp: August 27, 2023, 05:33:57 pm
content: Quote from: Karel on August 27, 2023, 03:35:31 pmIt's not that weird. Stdio is really geared towards streaming linearly through a file which has always worked fine with any size file. Likewise you can easily append to a file regardless of size. Consider most of the standard unix file utils for which the C standard library was originally designed: most of them never seek at all or possibly only to the end of the file. Also remember that for text files fseek with a non zero offset isn't even guaranteed to work at all.In my experience people doing random access to binary files they tend to use lower level IO calls such as read() / write() or mmap() anyway. They don't need or want the buffering support of stdio and there is little advantage to it. Those are OS dependent and come in 64 bit versions whenever the OS supports it.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 08:04:45 am
content: Quote from: ejeffrey on August 27, 2023, 05:33:57 pmCorrect. I showed a mmap() approach for a terabyte-sized file back in2011 at linuxquestions.org, but it does require a 64-bit POSIX-y (Linux, BSD, etc.) OS.Quote from: Karel on August 27, 2023, 12:22:35 pmSure. There is a standard, POSIX.1 aka IEEE Std 1003., for low-level OS interfaces in C, including I/O. It is mostly compatible with the C standard, but overrides or redefines certain details.You can find theSingle UNIX Specification at opengroup.org for free, whose Base specification is technically identical to POSIX.1; specifically,IEEE Std. 1003-2017 Base specificationis identical to POSIX.1-2017, and is the most recent and current version as of 2023.The relevant interfaces here areopen()/openat()for opening file-like things usingfile descriptors,close()for closing open file descriptors (and sockets),pread()for reading binary data from a specific offset without affecting the current file position,pwrite()for writing binary data starting at a specific offset without affecting the current file position, among other things.The type used for file descriptors isint; the type for amounts of bytes to be read/written issize_t; the type returned by read/write operations isssize_t; and the type describing a file position isoff_t. For their exact definitions, see thePOSIX.1/SUSv Base specification Headers section; specifically<sys/types.h>for a summary, and<stdlib.h>and<unistd.h>for where they are defined (and which you need to include to have these types available).Per POSIX.1,size_tis a type large enough to describe the size of anything in memory. It is the type thesizeofoperator evaluates to, for example.ssize_tis a type that can represent the size of anything in memory, and -1 (for error indication).The type relevant for this thread,off_t, can be either 32-bit or 64-bit in several 32-bit architectures, including x86.For the GNU C Standard Library akaglibc, the preprocessor macro_FILE_OFFSET_BITScan be used to set the size (by defining it to value64before including any headers) on 32-bit systems to select 64-bit "large file" support. Note that Linux is not perfectly POSIX in all details, so comparing the POSIX definitions to Linux and other systems' implementations using e.g.Linux man pages onlineis often useful.On OpenBSD and FreeBSD,off_thas been 64-bit for at least 15 years –– probably longer; it was 64-bit in 2008 –– even on 32-bit architectures.Fornewlib, the commonly used embedded standard C library on 32-bit microcontrollers and embedded processors, the default interface is assumed to be 32-bit, and if the 64-bit variants are supported they have different names (basically with64suffix) and use_off64_tfor the file position instead of_off_t.For example, an ANSI C program expanded fromCode:#define _FILE_OFFSET_BITS 64#include <stdlib.h>#include <unistd.h>#include <sys/types.h>#include <sys/stat.h>#include <fcntl.h>#include <errno.h>int main(int argc, char *argv[]){return 0;}that usesopen(),close(),read()/pread()and/orwrite()/pwrite(), optionallylseek(), and so on, will have a 64-bitoff_ttype on Linux, Mac OS, OpenBSD, FreeBSD, and several other operating systems, and is fully able to manipulate files over 4 GiB in size.Note that you should never rely onlseek()to obtain the file size. Usefstat()instead; but only if you cannot simply read everything until end of input (read()orpread()returning zero). The only fully portable and reliable way is to simply read until there is no more data to be read, aslseek()will fail if the file is appended to during reading, and neitherlseek()norfstat()work on pipes, sockets, character devices, or pseudofiles (like/procand/syspseudofile hierarchies in Linux that export OS/kernel information).

author: Karel
author_rank: Super Contributor
timestamp: August 29, 2023, 08:05:08 am
content: Quote from: TheCalligrapher on August 27, 2023, 04:10:18 pmThat poor design is dictated and/or left open to the implementation by ANSI C (ISO/IEC 9899:2011 (E)), for example:int fseek(FILE *stream, long int offset, int whence);where a long int usually has a size of 32 bits on 32-bit systems and 64 bits on 64-bit systems(ofcourse windows is the exception where it's usually 32 bits on both 32 and 64-bit versions of windows,because it's microsoft, they do everything different than the rest of the world).andint fsetpos(FILE *stream, const fpos_t *pos);where fpos_t is defined by ANSI C as:"a complete object type other than an array type capable of recording all theinformation needed to specify uniquely every position within a file."My point is, and probably I'm a bit naive here, I thought that ANSI C was created so that the same code can run correctlyafter being compiled whit whatever ANSI C compatible compiler.So, why did ANSI not add new functions like e.g. fseek64() and fsetpos64()?That would remove some burden when writing portable software that can also open large files on 32-bit systems.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 08:22:40 am
content: Quote from: Karel on August 29, 2023, 08:05:08 amBecause that, and other related stuff, moved (eventually) to POSIX.1 aka IEEE Std. 1003.1.Do remember that C standardization efforts moved from ANSI toISOin the 1989. ANSI started the standardization work in 1983 (X3J11 committee), and finalized it in 1989 as ANSI X3.159-1989. The effort then moved to ISO, which ratified the same ANSI X3.159-1989 standard as ISO/IEC 9899:1990. Those two are referred to as C89 (and C90, respectively).The first IEEE Std. 1003.1 (POSIX.1) was ratified in 1988. It was always about OS level (and initially UNIX and UNIX-like interoperable systems), so it made sense for ISO and IEEE C-related efforts to split into system-agnostic (ISO C) and system/OS-internal (POSIX.1) approaches.I and others could describe some of the history related to these standardization efforts (includingANSI escape codes, a subset of which are still commonly used to add color to and control text-based terminals, and were originally supported even in MS-DOS usingansi.sys, but whose support Microsoft dropped at some point from Windows' default terminal application), and indeed Microsoft has been the biggest antagonist with its historical Embrace-Extend-Extinguish (or Embrace-Extend-make-incompatible-profit!) approach. But, while historically accurate, it would sound a lot like ranting againt Microsoft, which actually has shifted its attitudes quite a bit in the last decade or so; however, you still cannotfullyrely on any standards when using C on Windows. For example, for no reason in particular, getting wide output or UTF-8 character set to work in the default console (command-line) still requires nonstandard extensions to be used.

author: DiTBho
author_rank: Super Contributor
timestamp: August 29, 2023, 10:11:05 am
content: edit: OTQuote from: Karel on August 29, 2023, 08:05:08 amI wrote a custom b*tree-based filesystem for XINU/Kernel, and that's exactly the approach I used:- my own lib-c library- my own lib-fs library with 32 and 64 bit versions of all the filesystem primitives!- practically { lib-fs , lib-fs32 , lib-fs64 }on GNU/Linux 32bit ... I have to use "magic" #defines, #include files in the right order and after the "magic" #define to be correctly #ifdef activated, and "magic" flags to have 64bit filesystem support (ext2,ext3,ext4,xfs do support it, fat16 and old minix-fs-v1 do not) on a 32bit userland.that's life ...

author: DiTBho
author_rank: Super Contributor
timestamp: August 29, 2023, 10:28:43 am
content: edit: OTor, do you prefer this approach?Code:void foo(string_t file_name,file_mode_t file_mode){fs_t fs;...fs'bit=64; /* please, use 64bit filesystem primitives */fs.open(file_name, file_mode);...}(my-c code)fs_t is an object and can overload all the filesystem methods { open, seek, pos, read, write, close, ... } according to the defined "bit" property.experimental personal-compiler project, I haven't taken an official position yet, in some ways I like it, in others I don't

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 10:58:55 am
content: Quote from: DiTBho on August 29, 2023, 10:28:43 amBe careful, there is "ANSI C" in the thread title, and OP already stated they're specifically interested in ANSI C solutions only!I should probably also note that POSIX.1 is all about interoperability between systems. The OS-level C interfaces are just one part of it; it also specifies a large set of command-line tools and a POSIX shell language derived from Bourne shells.TheBase definitionsdefine what POSIX.1 requires of the implementation (OS, C compiler, and so on). Basically, any implementation and its C compiler and standard C library can be POSIX.1 compatible, but an ANSI or ISO C standard implementation is not necessarily POSIX.1 compatible. POSIX.1 is much stricter, and defines things the C standards leave implementation defined, or do not touch at all.TheSystem Interfacesdefine the C language OS or system interfaces.TheShell and Utilities, Shell Command Languagedefines the POSIX shell command language. Bash does have a POSIX shell compatibility mode, but Debian Almquist SHell akadashis used when a faster (lower startup latency, smaller memory footprint) POSIX shell interpreter is needed;dashis included in BusyBox, and used in many embedded Linux targets like OpenWRT.The Rationale is informative, but itsPortability Considerationssection is very useful for anyone intending to write portable code.To simplify a bit: if we exclude Windows (but include WSL and WSL2, basically), POSIX.1 providesthestandard for interoperability under fully-featured operating systems,regardlessof the native word size. In fact, it would be perfectly possible to implement 64-bit file I/O POSIX.1 support even on an 8-bit or 16-bit processor. This is also the reason why many embedded targets use a freestanding C environment –– i.e., NO standard C library available at all, as specified in the ISO C standard(s) –– but a lot of POSIX.1'isms, especially in the types and how they are used. In particular, the way thatlogical typeslike file offset are given their own types whose properties are separately specified, instead of just hardcoding them aschar,short,int, orlong, the basic integer types defined by ANSI C. (It's not out of fanboyism to POSIX.1, but because POSIX.1 evolved over a couple of decades to betheway people can do interoperable C, surviving the competition among many different approaches; i.e. known working approaches. It's not perfect by any means, but it is the best we have right now.)

author: Karel
author_rank: Super Contributor
timestamp: August 29, 2023, 11:35:28 am
content: Long story short, if I want to write portable code that allows me to open large files also on 32-bit systems,I have to declare something like this:Code:#if defined(__APPLE__) || defined(__MACH__) || defined(__APPLE_CC__) || defined(__HAIKU__) || defined(__ANDROID__)#define fopeno fopen#else#define fseeko fseeko64#define ftello ftello64#define fopeno fopen64#endif#ifdef _WIN32#ifndef __MINGW32__/* needed for visual c */#undef fseeko#define fseeko _fseeki64#undef ftello#define ftello _ftelli64#undef fopeno#define fopeno fopen#endif#endifUsing this I can use fopeno(), fseeko() and ftello() on most common OS'es and with most common compilers, right?

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 11:56:33 am
content: Quote from: Karel on August 29, 2023, 11:35:28 amWon't work inglibc-based systems (many/most Linux ones) without modifications.It'd make more sense to create a header file, sayposix-io.hthat includes the necessary headers on target systems (usingpre-defined compiler macros), exposingfopen(),fclose(),errnoandstrerror(),ftello(),fseeko(),fgetc(),feof()(for examining the"already encountered end of input" flag, not testing whether file position is at the end of input),fread(),fwrite(),getline() and getdelim(), and the always 64-bitoff_ttype. For Windows, MSVCRT does not implementgetline()orgetdelim(), so one must provide fgets()/fgetc()/fscanf()-based compatible implementations (which perform acceptably well).Note that such an interface would be compatible with POSIX.1-2001 and later.Optionally it could also implement utility functions likeint get_file_size(FILE *, off_t *)(returning 0 if success and off_t saved, otherwise -1 witherrnoset) andint ftype(FILE *)(returning an int enum, for example 0 = unknown, 1 = regular file, 2 = pipe or socket or device (cannot obtain length, etc., with -1 indicating an error anderrnoset).

author: Karel
author_rank: Super Contributor
timestamp: August 29, 2023, 12:24:45 pm
content: Quote from: Nominal Animal on August 29, 2023, 11:56:33 amI tried and it seems to work fine on both Linux and windows with different compilers.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 12:29:46 pm
content: Quote from: Karel on August 29, 2023, 12:24:45 pmEnable compiler warnings, then. Or you didn't showallthe#defines and#includes you used.

author: Karel
author_rank: Super Contributor
timestamp: August 29, 2023, 01:00:58 pm
content: Quote from: Nominal Animal on August 29, 2023, 12:29:46 pmCode:#include <stdio.h>#include <stdlib.h>#include <string.h>#include <time.h>In the makefile:Code:CC = gccCFLAGS = -O2 -Wall -Wextra -Wshadow -Wformat-nonliteral -Wformat-security -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCELDLIBS = -lmSorry, I forgot to mention the compiler options

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 01:55:16 pm
content: Okay – although I'd add-D_FILE_OFFSET_BITS=64too as _LARGEFILE_SOURCE is obsolete for glibc –, but I still think you'd be better off using the POSIX.1 subset, using the same preprocessor logic. Something along the lines ofCode:#ifndef POSIX_IO_H#define POSIX_IO_H// Define USE_CUSTOM_GETLINE to use custom getline() implementation#undef NEED_CUSTOM_GETLINE// Define USE_CUSTOM_GETDELIM to use custom getdelim() implementation#undef NEED_CUSTOM_GETDELIM// Operating systems:#if defined(_WIN32)#if defined(__MINGW32__)// 32-bit MingW on Windows// TODO#elif defined(__MINGW64__)// 64-bit MingW on Windows// TODO#elif defined(_WIN64)// 64-bit Windows// TODO#else// 32-bit Windows// TODO#endif// Microsoft Visual C Runtime does not implement getline(), getdelim()#define NEED_CUSTOM_GETLINE#define NEED_CUSTOM_GETDELIM#elif defined(__APPLE__) && defined(__MACH__)// Mac OS// At least Darwin 10.5 and later use 64-bit off_t by default#ifndef _POSIX_C_SOURCE#define _POSIX_C_SOURCE 200809L#endif#include <stdlib.h>#include <stdio.h>#include <string.h>#include <errno.h>#elif defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__bsdi__) || defined(__DragonFly__)// BSD variants// Always 64-bit off_t, with fseeko() and ftello() defined in sys/types.h#include <stdlib.h>#include <sys/types.h>#include <stdio.h>#include <string.h>#include <errno.h>#elif defined(__HAIKU__)// Haiku// POSIX.1 compatible by default, and Haiku and BeOS off_t was always 64-bit.#include <stdlib.h>#include <stdio.h>#include <string.h>#include <errno.h>#elif defined(__linux__)// Linux and Android// Legacy definitions#ifndef _LARGEFILE_SOURCE#define _LARGEFILE_SOURCE#endif#ifndef _LARGEILE64_SOURCE#define _LARGEFILE64_SOURCE#endif// Ensure 64-bit off_t#undef _FILE_OFFSET_BITS#define _FILE_OFFSET_BITS 64// Include POSIX.1 C interfaces#ifndef _POSIX_C_SOURCE#define _POSIX_C_SOURCE 200809L#endif#include <stdlib.h>#include <stdio.h>#include <string.h>#include <errno.h>#endif /* Operating systems */#if defined(NEED_CUSTOM_GETLINE) || defined(USE_CUSTOM_GETLINE)// Custom getline() implementationstatic ssize_t custom_getline(char **lineptr, size_t *size, FILE *stream){// ...}#undef getline#define getline custom_getline#endif#if defined(NEED_CUSTOM_GETDELIM) || defined(USE_CUSTOM_GETDELIM)// Custom getdelim() implementationstatic ssize_t custom_getdelim(char **lineptr, size_t *size, int delim, FILE *stream){if (delim == '\n')return custom_getline(lineptr, size, stream);// ...}#undef getdelim#define getdelim custom_getdelim#endif#endif /* POSIX_IO_H */but the above is untested.This way, your source code only needs to start with#include "posix-io.h", and use the POSIX.1 interfaces (ftello(), fseeko(), and otherwise standard C99 or later <stdio.h>, <string.h>, and <errno.h> stuff), and you can omit the-Dcommand-line options. This then also works withclang, too, as well as other C compilers (including Intel Compiler Collection), and possibly even MS C++ in C "mode".

author: westfw
author_rank: Super Contributor
timestamp: August 29, 2023, 06:35:16 pm
content: QuoteAre you sure that your "32bit system" OPERATING SYSTEM and DISK FORMAT support "large files"?For instance, FAT32 is limited to 4GB-1, and HFS apparently 2GB.