bash-scripts, how to handle files with spaces in the names? - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: November 04, 2021, 04:29:08 pm
content: I have troubles handling bash-scripts because some filenames contain spaceslike theseCode:Borland Makefiles.rstCodeBlocks.rstCodeLite.rstEclipse CDT4.rstGreen Hills MULTI.rst <------- e.g. ---------- Green + space + Hills + space + MULTI.rstKate.rstMSYS Makefiles.rstMinGW Makefiles.rstNMake Makefiles JOM.rstNMake Makefiles.rstNinja Multi-Config.rstNinja.rstSublime Text 2.rstUnix Makefiles.rstVS_TOOLSET_HOST_ARCH.txtVisual Studio 10 2010.rstVisual Studio 11 2012.rstVisual Studio 12 2013.rstVisual Studio 14 2015.rstVisual Studio 15 2017.rstVisual Studio 16 2019.rstVisual Studio 6.rstVisual Studio 7 .NET 2003.rstVisual Studio 7.rstVisual Studio 8 2005.rstVisual Studio 9 2008.rstWatcom WMake.rstXcode.rstI am working with a python text-based database where each line is a db-entry and looks like thisCode:object package path/filename md5sum sizeso for mekind = line_get_arg1(line) (it can be object, symbolic link, etc)package = line_get_arg2(line)pfile = line_get_arg3(line) <------------ this only works if the filename doesn't contain spacesThe database was created by a python engine, which is somehow able to handle files with spaces in the names, but I cannot use the python engine on this machine, I can only use bash or a C-written program.

author: borjam
author_rank: Unknown
timestamp: November 04, 2021, 04:31:44 pm
content: You can surround the names with quotes or you can prepend a backslash to each space.

author: PKTKS
author_rank: Super Contributor
timestamp: November 04, 2021, 04:55:41 pm
content: In PERL it is already built in and has at least 5 ways ..ps please fix that ... this for quoted - arbitraryqq ('fo%o$ bar not a easy method. $$$ ');Use this operator instead for proper escapingquotemeta('fo%o$ bar not a easy method. $$$ ');In python requires external "module" thingy..Code:import rere.escape('fo%o$ bar not a easy method. $$$ ')PaulPS alas if not entirely clear you can use that in a PIPE over bash...

author: Benta
author_rank: Super Contributor
timestamp: November 04, 2021, 06:45:06 pm
content: In bash, file names with spaces are surrounded by single quotes.

author: DiTBho
author_rank: Super Contributor
timestamp: November 04, 2021, 07:35:08 pm
content: yes, but ... I have to "extract" the filename from a single line of text read from a file.

author: DiTBho
author_rank: Super Contributor
timestamp: November 04, 2021, 07:38:42 pm
content: Code:object package path/filename md5sum sizeobject package path/filename md5sum sizeobject package path/filename md5sum size...object package path/filename md5sum sizeobject package path/filename md5sum sizeobject package path/filename md5sum sizeCode:while IFS= read -r lineadolinea_get_arg1 $lineakind = "$ans"linea_get_arg2 $lineapackage = "$ans"linea_get_arg3 $lineapfile = "$ans" # this only works if the filename doesn't contain spacesdone < $source

author: Ian.M
author_rank: Super Contributor
timestamp: November 04, 2021, 07:39:56 pm
content: If you cant modify the python database engine to output the file with the filenames quoted, or with the spaces escaped in a way that bash can handle, then you'll have to deal with the file as-is. Fortunately it isn't ambiguous, as the filename is always followed by two more parameters, so the algorithm required would be to take the whole line and locate the separators before and after the filename, working back from the end of the line for the latter then slice out the string you need.How to do that in bash is another matter. I suspect it would be better to use an external utility such as grep or sed rather that trying to wrangle it with bash's native string handling.

author: golden_labels
author_rank: Super Contributor
timestamp: November 04, 2021, 07:41:42 pm
content: Bash doesn’t care if there are spaces in file names. In fact it doesn’t care about any byte at all, except for the byte 0, which couldn’t be handled by utilities based on c-strings anyway. You may even have newlines in a file name. If there are any issues with spaces, it’s nearly always because of using improper variables expansion (without surrounding them with double quotes) or using wrong tools (likelsto list files).Therefore please provide the actual,minimalcode example, that depicts your issue. So far the answer from me would be “it works here”.The data format you have mentioned is already broken by design, though, as it will not be able to handle newlines. But with the constraints given, there is no reasonable way to address that, so I am only mentioning the problem.

author: DiTBho
author_rank: Super Contributor
timestamp: November 04, 2021, 07:55:28 pm
content: Quote from: golden_labels on November 04, 2021, 07:41:42 pmthe above codeon a text file like thisCode:object package-x Green Hills MULTI.rst md5sum sizeobject package-x NMake Makefiles JOM.rst md5sum sizeobject package-x Visual_Studio_14_2015.rst md5sum sizeobject package-x Visual Studio 14 2015.rst md5sum size(source.txt)Full minimal code with service functionsCode:function line_get_arg1(){ans="$1"}function line_get_arg2(){ans="$2"}function line_get_arg3(){ans="$3"}function line_get_arg4(){ans="$4"}source="source.txt"while IFS= read -r linedoline_get_arg1 $linekind="$ans"line_get_arg2 $linepackage="$ans"line_get_arg3 $linepfile="$ans" # this only works if the filename doesn't contain spacesecho "--------------------------"echo "kind = [$kind] "echo "pack = [$package] "echo "file = [$pfile] "done < $sourceit outputsCode:--------------------------kind = pack = [package-x]file =  <-------------------------- wrong!!!!--------------------------kind = pack = [package-x]file =  <-------------------------- wrong!!!!--------------------------kind = pack = [package-x]file = [Visual_Studio_14_2015.rst] <-------------------------- correct!--------------------------kind = pack = [package-x]file =

author: golden_labels
author_rank: Super Contributor
timestamp: November 04, 2021, 08:03:34 pm
content: Code:$ cat readEntries#!/usr/bin/env bashwhile read object package tail; dounset fnamehash=size=for next in $tail; doif [[ -z "$fname" ]]; thenfname="$hash"elsefname+=" $hash"fihash="$size"size="$next"doneecho '=== Entry ==='printf ' Object: “%s”\n' "$object"printf ' Package: “%s”\n' "$package"printf ' File name: “%s”\n' "$fname"printf ' Hash: %s\n' "$hash"printf ' Size: %s\n\n' "$size"done$ cat inputsobj#9 pkg-1 Borland Makefiles.rst cdd7ae77e9686e441054887668690f3f 123004obj#2 pkg-3 CodeBlocks.rst d8cb14f07836f07874f2ae11b27e095e 98941obj#4 pkg-4 Green Hills MULTI.rst <------- e.g. ---------- Green + space + Hills + space + MULTI.rst 7d99181a2b9b9e533c1470958e343d16 1029394obj#7 pkg-12 VS_TOOLSET_HOST_ARCH.txt ed60c6dc919f27ebd107041a3b16915d 149951$ ./readEntries <inputs=== Entry ===Object: “obj#9”Package: “pkg-1”File name: “Borland Makefiles.rst”Hash: cdd7ae77e9686e441054887668690f3fSize: 123004=== Entry ===Object: “obj#2”Package: “pkg-3”File name: “CodeBlocks.rst”Hash: d8cb14f07836f07874f2ae11b27e095eSize: 98941=== Entry ===Object: “obj#4”Package: “pkg-4”File name: “Green Hills MULTI.rst <------- e.g. ---------- Green + space + Hills + space + MULTI.rst”Hash: 7d99181a2b9b9e533c1470958e343d16Size: 1029394=== Entry ===Object: “obj#7”Package: “pkg-12”File name: “VS_TOOLSET_HOST_ARCH.txt”Hash: ed60c6dc919f27ebd107041a3b16915dSize: 149951Note that this can’t deal with the aforementioned shortcomings of the data format. Itwillfail for some inputs. If possible, I would strongly recommend:Using a properly structured data storage format, like JSON or XML.Using proper extraction/manipulation tools, likejqor XML processors.Switching to some sane language, like Python or Groovy. Bash is unsuitable for data processing.

author: DiTBho
author_rank: Super Contributor
timestamp: November 04, 2021, 09:09:16 pm
content: @golden_labelsPerfect! Your approach worked so well that I converted the entire database and I also learned new things!Thank you very much

author: PKTKS
author_rank: Super Contributor
timestamp: November 05, 2021, 10:55:28 am
content: Quote from: golden_labels on November 04, 2021, 08:03:34 pmNice bashing..but true that it will fail when characters other than the page code being use are present...Had a ton of issues converting from LATIN charset to plain ASCII using not only bash but also SQL itself. Usually the quotemeta operator solves the problem - just need to really quote the nasty chars UTF or whatever present.JSON is not a problem as long as you decode it properly escapedCode:# my simple way to load a properly escaped JSONuse JSON qw( decode_json );@lines = <FLE>;quotemeta($decoded) = decode_json("@lines");Paul

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 05, 2021, 12:08:42 pm
content: If your Bash script is not localized, addexport LANG=C LC_ALL=Cnear the beginning, so that input is considered a sequence of bytes, instead of sequence of characters. This way there is no illegal input.If the input may contain leading or trailing whitespace, or multiple consecutive spaces as field separators, or use tabs instead of spaces or even a mix, then I'd suggest a more complicated, but also much more generic variant:Code:Separator=$'[ \t]'# Strip VarNamefunction Strip() {local -n dst="$1"local olddst="${dst##$Separator}"local newdst="${olddst%%$Separator}"while [[ ${#newdst} != ${#olddst} ]]; doolddst="${newdst##$Separator}"newdst="${olddst%%$Separator}"donedst="$newdst"}# FirstField SrcVarName DstVarNamefunction FirstField() {local -n src="$1" dst="$2"dst="${src%%$Separator*}"if [[ ${#dst} == ${#src} ]]; thensrc=""elsesrc="${src#*$Separator}"local newsrc="${src##$Separator}"while [[ ${#newsrc} != ${#src} ]]; dosrc="$newsrc"newsrc="${src##$Separator}"donefi}# LastField SrcVarName DstVarNamefunction LastField() {local -n src="$1" dst="$2"dst="${src##*$Separator}"if [[ ${#dst} == ${#src} ]]; thensrc=""elsesrc="${src%$Separator*}"local newsrc="${src%%$Separator}"while [[ ${#newsrc} != ${#src} ]]; dosrc="$newsrc"newsrc="${src%%$Separator}"donefi}TheSeparatoris either a single separator, or the list of acceptable separators in square brackets.Given a variable name,Stripremoves any leading or trailingSeparators.Given a source and destination variable names,FirstFieldandLastFieldsplit off the field from the source variable into the destination variable, and update the source variable.In the Bash functions, thelocal -n foo="$1"makesfooa nameref-attributed variable: referring to it or modifying it actually refers to or modifies the variable named in the first parameter to the function. Nifty.Using the above, the loop isCode:while read Record ; doStrip RecordFirstField Record ObjectFirstField Record PackageLastField Record SizeLastField Record MD5SumPathname="$Record"printf 'Object=%s, Package=%s, Pathname=%s, MD5Sum=%s, Size=%s\n' "$Object" "$Package" "$Pathname" "$MD5Sum" "$Size"doneFor example, given (using\tto denote an embedded tab),\tobj#4 pkg-4 \t Green Hills MULTI.rst\te.g. Green + space + Hills + space + MULTI.rst 7d99181a2b9b9e533c1470958e343d16 1029394the above will outputObject=obj#4, Package=pkg-4, Pathname=Green Hills MULTI.rst\te.g. Green + space + Hills + space + MULTI.rst, MD5Sum=7d99181a2b9b9e533c1470958e343d16, Size=1029394i.e., handle correctly all linear whitespace (listed in$Separator) as a single field separator.In the general case, you useFirstFieldandLastFieldto "prune" the non-whitespace-containing fields off the input record, so that whatever is left is stored in the possibly-whitespace-containing field.The easier option, however, and the one I would recommend, is to simply prepend each tab or space that is not a separator with a backslash. For example, givenObject Package Path\ or\ file\ name MD5Sum Sizea simpleIFS=$' \t'while read Object Package Pathname MD5Sum Size ; doprintf 'Object=%s, Package=%s, Pathname=%s, MD5Sum=%s, Size=%s\n' "$Object" "$Package" "$Pathname" "$MD5Sum" "$Size"doneoutputsObject=Object, Package=Package, Pathname=Path or file name, MD5Sum=MD5Sum, Size=Sizebecause by default, the Bashreadbuilt-in supports escaping tabs and spaces by prepending a backslash to them. (Prepending a backslash before a newline is a line continuation: the Bashreadbuilt-in includes the continued lines in the record, but removes the backslash-newline pairs.)If the pathnames are generated by a Bash script, you can use the following function to add those backslashes to any shell variables; just call the function with the names of those variables as parameters:Code:function Backslashes() {local tab=$'\t'while [[ $# -gt 0 ]]; dolocal -n dst="$1"dst="${dst//\\/\\\\}"dst="${dst// /\\ }"dst="${dst//$tab/\\$tab}"shift 1done}For example, if you haveFOO="Foo has spaces"andBAR=$"Bar\thas\t\ttabs", thenBackslashes FOO BARmodifiesFOOtoFoo\ has\ \ spaceandBARto$"Bar\\\thas\\\t\\\ttabs"(i.e., adds backslashes before spaces and tabs). This means that newlines cannot be escaped using this method, unfortunately.For full support of actual backslash escape sequences (\0 \a \b \t \n \v \f \r \\ \' \" \ooo \xHH \uHHHH \UHHHHHH) and quoting (single- and double-quotes, with backslash escapes only parsed outside quotes and within double quotes), you can either use a slow and complicated Bash function, or preprocess the data using e.g. awk, into a stream of nul-separated fields. This does require a fixed number of output fields per record, but that way all possible paths and file names in Linux are supported, and your Bash script can handle all possible C strings (i.e., everything except strings with embedded NUL bytes) as fields.For example, if the input contains five nul-separated fields (Object Package Pathname MD5Sum Size) per record, the following Bash loop,Code:while read -rd "" Object ; doread -rd "" Package || breakread -rd "" Pathname || breakread -rd "" MD5Sum || breakread -rd "" Size || break# Object Package Pathname MD5Sum Sizedonewill read them very efficiently, and is not affected byIFS.A more flexible option (not requiring a fixed number of fields per record) is to use a filter, say written in C, that converts the input into a stream of records. I suggest using ASCII US (31, unit separator) for field separator, ASCII RS (30, record separator) for record separator, and ASCII SUB (26, substitute) following a single digit ('0' for NUL, '2' fur SUB, '6' for RS, and '7' for US) for substitutes ("escape sequences"), so that the following Bash loop can parse such records very efficiently:Code:oldIFS="$IFS" ; IFS=$'\037'while read -rd $'\036' -a Fields ; dofor ((i = ${#Fields[@]} - 1; i >= 0; i--)) ; dotemp=$'${Fields//\0326/\036}' # Record separatorstemp=$'${temp//\0327/\037}' # Field separatorstemp=$'${temp//0320/}' # Skip embedded NULs since Bash can't handle them anywayFields=$'${temp//\0322/\032}' # Substitutesdone# Use Fields ...doneIFS="$oldIFS"Except for embedded NUL bytes, this provides you with an array corresponding to the exact fields in each record.

author: golden_labels
author_rank: Super Contributor
timestamp: November 05, 2021, 08:08:51 pm
content: Quote from: PKTKS on November 05, 2021, 10:55:28 amIt will fail in much simpler situations: newlines, U+000D, U+0009, anything interpretable as a glob pattern, possibly also with specific file names containing very large number of space-separated components. I think I might have missed a half of other things. One may attempt to solve each of those issues, but then new edge cases appear and we end up with an unmaintainable monstrosity that can’t even be easily verified to actually work. Which is why I haven’t attempted fixing it and, instead of addressing shortcomings, resolved to adding a clear warning and a suggestion to use tools that can actually deal with the problem properly.

author: DiTBho
author_rank: Super Contributor
timestamp: November 05, 2021, 08:33:00 pm
content: With your help, guys, yesterday I converted all the files(1) with filter that replace ' ' with '_', the data base got updated accordingly.Problem solved (for now)(1) 2194 files