Unix & Linux: How can I remove the BOM from a UTF-8 file? (7 Solutions!!)
Channel: Roel Van de Paar
Date: 12/08/2020

Unix & Linux: How can I remove the BOM from a UTF-8 file?


The Question: I have a file in UTF-8 encoding with BOM and want to remove the BOM. Are there
any linux command-line tools to remove the BOM from the file?
$ file test.xml
test.xml: XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long
lines

Solutions: Please watch the whole video to see all solutions, in order of how many people found them helpful

== This solution helped 29 people ==
It is possible to remove the BOM from a file with the tail command:
tail -c +4 withBOM.txt ＞ withoutBOM.txt

== This solution helped 77 people ==
A BOM doesn't make sense in UTF-8. Those are generally added by mistake by
bogus software on Microsoft OSes.
dos2unix will remove it and also take care of other idiosyncrasies of Windows
text files.
dos2unix test.xml

== This solution helped 87 people ==
If you're not sure if the file contains a UTF-8 BOM, then this (assuming the
GNU implementation of sed) will remove the BOM if it exists, or make no changes
if it doesn't.
sed '1s/^xEFxBBxBF//' ＜ orig.txt ＞ new.txt
You can also overwrite the existing file with the -i option:
sed -i '1s/^xEFxBBxBF//' orig.txt
If you are using the BSD version of sed (eg macOS) then you need to have bash
do the escaping:
 sed $'1s/xefxbbxbf//' ＜ orig.txt ＞ new.txt

== This solution helped 5 people ==
You can use
LANG=C LC_ALL=C sed -e 's/r$// ; 1 s/^xefxbbxbf//' -i -- filename
to remove the byte order mark from the beginning of the file, if it has any, as
well as convert any CR LF newlines to LF only. The LANG=C LC_ALL=C tells the
shell you want the command to run in the default C locale (also known as the
default POSIX locale), where the three bytes forming the Byte Order Mark are
treated as bytes. The -i option to sed means in-place. If you use -i.old, then
sed saves the original file as filename.old, and the new file (with the
modifications, if any) as filename.
----
I personally like to have this as ~/bin/fix-ms; for example, as
#!/bin/dash
export LANG=C LC_ALL=C
if [ $# -gt 0 ]; then
for FILE in "$@" ; do
sed -e 's/r$// ; 1 s/^xefxbbxbf//' -i -- "$FILE" || exit 1
done
else
exec sed -e 's/r$// ; 1 s/^xefxbbxbf//'
fi
so that if I need to apply this to say all C source files and headers (my old
code from the MS-DOS era, for example!), I just run
find . -name '*.[CHch]' -print0 | xargs -r0 ~/bin/ms-fix
or, if I just want to look at such a file, without modifying it, I can run
~/bin/ms-fix ＜ filename | less
and not see the ugly ＜U+FEFF＞ in my UTF-8 terminal.

== This solution helped 33 people ==
**** Using VIM ****
 1. Open file in VIM:
 vi text.xml
 2. Remove BOM encoding:
 :set nobomb
 3. Save and quit:
 :wq

Recently I found this tiny command-line tool which adds or removes the BOM on
arbitary UTF-8 encoded files: https://code.google.com/p/utf-bom-utils/ (new
link at github)
Little drawback, you can download only the plain C++ source code. You have to
create the makefile (with http://www.cmake.org/, for example) and compile it by
yourself, binaries are not provided on this page.

With thanks & praise to God, and with thanks to the many people who have made this project possible! | Content (except music & images) licensed under cc by-sa 3.0 | Music: https://www.bensound.com/royalty-free-music | Images: https://stocksnap.io/license & others | With thanks to user Wernfried Domscheit (https://unix.stackexchange.com/users/314876), user Trevor Murdock (https://unix.stackexchange.com/users/391916), user Stephane Chazelas (https://unix.stackexchange.com/users/22565), user Nominal Animal (https://unix.stackexchange.com/users/157724), user Michael Homer (https://unix.stackexchange.com/users/73093), user Matthew Buckett (https://unix.stackexchange.com/users/23145), user m13r (https://unix.stackexchange.com/users/80062), user Joshua Pinter (https://unix.stackexchange.com/users/99662), user CSM (https://unix.stackexchange.com/users/107535), and the Stack Exchange Network (http://unix.stackexchange.com/questions/381230). Trademarks are property of their respective owners. Disclaimer: All information is provided "AS IS" without warranty of any kind. You are responsible for your own actions. Please contact me if anything is amiss at Roel D.OT VandePaar A.T gmail.com.