Mystery process adding "options rotate" to resolv.conf file - Page 1
Forum section: electronics / general

author: peter-h
author_rank: Super Contributor
timestamp: January 11, 2025, 04:55:26 pm
content: I have a server, centos / nginx, whose content is just static html. No PHP, etc. No interactive stuff. No JS sent to the client browser.I have a guy who looks after some sites for me. He is pretty familiar with unix stuff and databases...The server has a cron job which pings a healthcheck site every hour so we can see if the server is down. The websites on it work but this ping stopped a month ago.Something is messing with the resolv.conf file. It was found to contain no nameserver entries (which presumably is why pinging hc-ping.com was failing) and contains some 80000 lines with "options rotate", one being added every 80 seconds.But there is nocronjob doing this. But there must be a "script" somewhere which is running. But where?The server is a virtual server and was originally set up about 20 years ago. It was updated to latest OS maybe 10 years ago.Access (updates) is done with rsync, from a local copy held on a PC. There is also sftp and ftp access.Does this ring a bell with anyone? It could be an attack (which failed to do much) or it could be some accident.

author: madires
author_rank: Super Contributor
timestamp: January 11, 2025, 08:57:06 pm
content: Could be some DHCP automation gone wild. A 'cd /etc' and 'grep -R "options rotate"' might find the source.

author: magic
author_rank: Super Contributor
timestamp: January 11, 2025, 09:03:10 pm
content: You could start with something like fuser or lsof to see if any process has this file opened constantly.Unfortunately, I don't know how to wait for some process to open a file and log which process it was.You could also make the file read only and see if it helps and see if any daemon logs errors because of it.I'm a big sys*^&d hater so it surely must be this POS gone rogue, somehow

author: peter-h
author_rank: Super Contributor
timestamp: January 11, 2025, 10:23:07 pm
content: Thank youA partial success: /etc/dhcp/dhclient-exit-hooks. Supposedly it runs whenver dhcp gets or loses IP address. Currently it does echo 'options rotate' >> /etc/resolv.conf. Now why would somebody have set this up?The file was set to R/O.Doesn't sound like a hacker because a hacker with that access could have done much more damage.The DNS IP disappeared from that file around 21 Nov 2024.Turns out that an image restore was performed on that server on that day. It was done to reverse some experimental work on directory listing privileges. Why the restored image buggered up that DNS IP file, is a mystery.Anyway, this has provided an excuse for some cleanup work, like log rotation, and removing no longer used services like pop boxes

author: Postal2
author_rank: Frequent Contributor
timestamp: January 11, 2025, 11:29:10 pm
content: Quote from: peter-h on January 11, 2025, 10:23:07 pmWhat "mysteries" can there be if you restore the system from an image and it does not boot after that.It's clear that toothbrushes and plunger are to blame for everything.

author: peter-h
author_rank: Super Contributor
timestamp: January 12, 2025, 07:34:14 am
content: Oh dear, ChatGPT a.k.a. Postal2 (from Russia) is back in action.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 12, 2025, 08:48:21 am
content: Quote from: peter-h on January 12, 2025, 07:34:14 amI wonder what kind of ChatGPT prompt one needs to use to get this output.

author: peter-h
author_rank: Super Contributor
timestamp: January 12, 2025, 08:55:19 am
content: IF the image operation was the cause, it is interesting how it could have caused it, because the virtual hosting company shuts down the server before doing the image backup.Or at least I think they do! Maybe they don't, because the site would go dead for the few mins it takes to do it. In that case it is not surprising that there are file system inconsistencies. I can also do a manual snapshot and for that I can do a shutdown.Same as doing a Trueimage backup for example. That does a "shadow copy" (whatever that is). Doing the backup from a boot DVD of TI would avoid that issue.

author: Postal2
author_rank: Frequent Contributor
timestamp: January 12, 2025, 09:25:16 am
content: Quote from: peter-h on January 12, 2025, 08:55:19 amRAID, second copy stops.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 12, 2025, 01:01:44 pm
content: Quote from: peter-h on January 12, 2025, 08:55:19 amI use LVM2 for filesystem snapshots, and that yields true filesystem state (including page cache, i.e. kernel-cached data; not just data on-disk) at that point in time without delaying future writes, because the snapshot is copy-on-write: the first write to the state copies that sector/chunk. As a result, one needs to keep some disk space in reserve for the snapshots, but that's about it. Note that it isn't just the data on disk; dirty pages in page cache are included in the snapshot, so it does not matter whether some parts of the filesystem state is in memory or not. Whether you runsyncbefore snapshotting or not, does not affect the contents of the snapshot.For the image to be consistent, you want the system to bequiescent: not actively modifying files, with all changes in userspace service/application/library memory buffers written to disk. Note that you only need it for a fraction of a second, for the duration ofcreatingthe snapshot using LVM2 tools; the snapshot itself remains usable afterwards, while the parent filesystem is in normal use. (There are some tricks, like creating a preliminary snapshot without quieting the system, then updating the snapshot when quiescent, to reduce the real-world impact of the snapshotting. Most users automate all this into scripts or maintenance panels.)Some services, like nginx and Apache, are easy to stop for the duration of the snapshot, to ensure quiescent state; some have dedicated options for this for userspace snapshotting. Some services, like systemd and associated services, are impossible to stop/sync/quieten this way without essentially rebooting the userspace, because they're not designed to support it.(In addition to experimenting with replacing the standard C library with something more suitable for today, especially including asynchronous I/O support, I've also investigated exactly what kind ofinit systemwould allow this, and monitoring service status, without the tendency and vulnerability to single point failures and security problems that systemd type of designs have. I've also done quite a lot of security-sensitive service development, although I'm no cryptographer (and rely on proper professionals on that front and many others), so if anyone is interested in discussing these further, just drop me a PM or an email, or start a new thread here. A particular detail is secure Nginx/Apache configuration, when using server-side scripts; also when having a complex hierarchy of web administrators, and their access controls via group membership and Unix access controls.)When using complete virtual hosts, the VM snapshot includes both the "processor" state and RAM, and the filesystem(s). When using local area storage with sufficient caching (in the gigabytes), a virtual image can be transferred from one hardware to another within a fraction of a second, limited only by the throughput of the local area network and the size of the RAM image and processor state. (Networked filesystem state does not need to be transferred. A single-processor 1GB image using 10G networking can be transferred within a couple of seconds, easily fast enough for not even TCP/IP connections to break, only a couple-second "hiccup" during transfer.) Because these snapshots includes the processor state and RAM, there is no need to make the system quiescent before snapshotting. The downside is the small overhead from running a virtualized OS instead of on the bare metal. When filesystem storage is network-connected also, and you don't use GPGPU or other external hardware computing accelerators, the difference to bare metal is basically neglible. (I'm not up to date how well Nvidia stuff is virtualized/virtualizable to a specific VM, though.)

author: Postal2
author_rank: Frequent Contributor
timestamp: January 12, 2025, 03:50:46 pm
content: Quote from: Nominal Animal on January 12, 2025, 01:01:44 pmI know from my own experience that if you are working in the VC++ 6.0 environment with a project and your program gets a blue screen, then you will be collecting pieces of your code from fragments of chains restored into separate files. That is, the files are open, saved, the system is in a state of rest. You interrupt the computer's power supply and suffer for a long time after that. Accordingly, having made an instant snapshot of the disk under these conditions, you will suffer similarly.There is a special file system for Linux:https://www.axllent.org/docs/btrfs-raid1/

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 12, 2025, 04:30:56 pm
content: Quote from: Postal2 on January 12, 2025, 03:50:46 pmNo. That may be how Windows works, or how VC++ 6.0 works under Wine, but it definitely isn't how Linux and applications in Linux work.You do need to issue asyncfs()on the mounted snapshot filesystem (sync -fany-file-on-the-mounted-snapshot) to ensure the snapshot data on the storage medium contains all cached data, but other than additional I/O load, it will not affect the use of the original filesystem.I do not use VC++ 6.0, so I do not know and have not investigated how it quietens its own state. All editors I've used in Linux do use temporary files, either in user-specific temporary directory (under/tmp/application-and-username/,/var/tmp/application-and-username/,~username/.application/, or~username/.cache/application-or-tempname/) or in a temporary subdirectory (name starting with a.to "hide" it from normal file listing, by convention) to the original file. This is not kept instantly up to date, but auto-saved on a varying frequency.All these editors fall into one of the two categories, similar to Nginx/Apache and systemd: either they quieten easily or automatically, or they do not at all. It is up to the application and service designer/developer to decide whether state is kept in recoverable files, or just in RAM. It is trivial to detect the loss of focus in graphical environments, and use a timeout for user input in any environment, as a trigger for flushing recoverable state to disk andfsync()it in a helper thread (not hindering the other threads of the process in any way). If state is only maintained in RAM –– which it often is, since it is easiest, and most developers don't care about weird use cases or snapshots or error situations ––, then you're better off running it in a Virtual Machine, and snapshotting that VM, as snapshotting just the filesystem will not capture the full state.Quote from: Postal2 on January 12, 2025, 03:50:46 pmwhich will do absolutely nothing about the fundamental problem here, and not perform any better wrt. filesystem snapshots compared to LVM2, assuming you use a journaling filesystem likeext4(default in most Linux distributions) and multi-device storage (RAID1/5/6/10 etc.).

author: Postal2
author_rank: Frequent Contributor
timestamp: January 12, 2025, 04:46:58 pm
content: Quote from: Nominal Animal on January 12, 2025, 04:30:56 pmTry turning off the power unexpectedly for the system or removing the battery from the laptop.Or, for the programmer, send zeros to the RAM controller registers.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 12, 2025, 05:49:20 pm
content: Quote from: Postal2 on January 12, 2025, 04:46:58 pmIt's happened to me often enough with my small Linux SBC's. I know very, very well how to deal with it in Linux. The only thing that tends to get harmed is if I have to use an SD card for the storage; they really don't like losing power in the middle of a write, generally speaking.What happens is that data already written to the filesystem journal (I use the default 30 second time limit) is autorecovered on the next boot, and data not written to disk yet is lost. Applications that use temporary files that they applyfsync()on will be able to recover from the crash.There is no scenario in Linux where application data would appear in random files on the filesystem. LVM2 does not affect this, nor does mdraid/dmraid/hardware RAID.Linux uses the inode model, where filesystem entries (file and directory names) are opaque byte sequences delimited by NUL (0), path components delimited by / (47), and each refers to an inode. An inode contains both the data and metadata (size, owner user and group, access mode, extended attributes). An anonymous temporary file is one where the filesystem entry ("name") has been deleted, but the file is open by at least one process. An unnamed temporary file is one opened using theO_TMPFILEflag, specifying only the directory where the unnamed temporary file may be named later using thelinkat()syscall, in Linux. If the operating system crashes, these will be lost; although a custom invocation offsckcan recover these into named files on many filesystems, including ext4. If recovered, these will have the contents that were written to disk (or journaled) before the OS crashed. In no case will the temporary file data be scattered or confused somehow.Windows, even NTFS, works very differently, so the behaviour between Linux and Windows (including WSL, and when running Windows applications in Linux using Wine), is very different.As to the resolv.conf mess, I would inspect the resolver part of systemd. Standard C library reads/etc/resolv.confat run time when certain POSIX functions are run, so network service managers often replace it dynamically whenever the configuration changes, keeping the actual configuration somewhere else. (Similarly fornsswitch.conffor name services in general, although services typically do not need to modify this one at run time.)If you write C code, in Linux you should use POSIXgetaddrinfo()for name-to-address translation, andgetnameinfo()for address-to-name translation, notgethostbyname()/gethostbyaddr(). The underlying issue is the same as open-coding opendir()/readdir()/closedir(), instead of using the POSIXnftw()orscandir()orglob(), the latter having been available in the same standard C libraries for decades. (Well, except that gethostbyname() and gethostbyaddr() are actually deprecated already nowadays.)

author: DiTBho
author_rank: Super Contributor
timestamp: January 12, 2025, 09:45:23 pm
content: @Nominal AnimalI have observed something similar in the past, in two occasionshuman error due to high stressI usually use the tab-key to select a file on the shell, one night I wrongly select the target of the "cat" commandand I only discovered the error weeks later, when I rebooted the router and found that a server was not starting as usual. Of course it wasn't, one of its configuration files had been overwritten with something that didn't make sense for that server, for example imagine putting "/system/config/ssmp" in /system/config/sshd" ...kernel bugThe HP C3600 workstation had a serious bug, I posted a patch years ago on the Linux ML.I remember, I was working on an experimental kernel and testing a PCI sATA HBA card I had just purchased on eBay. It passed all the tests I ran in 24 hours. Too bad there was a remote possibility of catastrophic failure, due to a bug involving both the HPPA2 CPU and the DMA subsystem, all of which in 99.99% of the cases do not happen, but in that 0.01% of probability, when it does happen... the consequences are catastrophic: in the case... parts of the content of files opened in RW ending up in the wrong place because the DMA wrote the wrong thing in the right place, and the kernel did not notice.The result was... that I opened /etc/profile, and found inside a piece of C code that I was working on in the terminal...Of course, being an experimental machine, ram-rootfs. So, no serious damage.

author: DimitriP
author_rank: Super Contributor
timestamp: January 12, 2025, 10:16:35 pm
content: There are few "mysteries" when it comes to "computers going wrong" and it usually involves someone else having done something that they have disclosed to no one other than "there is an issue that cropped up".As I was "reading the matrix" from what the OP stated, here is what was underlined, highlighted and blinking:QuoteQuoteQuoteQuoteQuoteQuoteSo far the short version of this is "suddenly it started doing wierd things"....and finally the beans are spilled:QuoteAha!QuoteSuuuuuure....Someone was screwing around, and instead of undoing their work -probably because they weren't sure they knew what they were doing in the firstplace- decided to "put it back" from "the image"Then a little bit of hell broke loose and they washed their hands of it because "I restored from the image - everything should be working" . Except it wasn't.And here we are.If all these events have the same person in common... you need a new person or keep your fingers crossed for the next time something similar goes wrong; "suddenly"!EOT(fixed spelling of someone , probably & because )

author: peter-h
author_rank: Super Contributor
timestamp: January 12, 2025, 10:46:29 pm
content: Hardly.

author: Postal2
author_rank: Frequent Contributor
timestamp: January 12, 2025, 11:17:48 pm
content: Quote from: Nominal Animal on January 12, 2025, 05:49:20 pmNow you know what's going on in Windows, the lost megabytes of code will be in the file123.chk files in small pieces (after checking the disk). And when I found these pieces for the first time, I was very happy.Quote from: DimitriP on January 12, 2025, 10:16:35 pmI would also draw attention to this.

author: 5U4GB
author_rank: Super Contributor
timestamp: January 13, 2025, 04:38:16 am
content: Just as a general observation, if something randomly changes your networking config behind your back and breaks things then 99% of the time it's systemd. They really, really like screwing around with people's networking configs.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 13, 2025, 07:53:32 am
content: Quote from: Nominal Animal on January 12, 2025, 05:49:20 pmI had this happen to me in Windows (XP, NTFS) over a decade ago: at some point an mp3 music file was corrupted such that it contained random excerpts of other mp3 files. Apparently the mp3 file stream format itself was robust enough that mp3 player software decoded it just fine with minor glitching. It became an interesting collection of music clips of 1-2 seconds each.Whether it was NTFS or Windows filesystem implementation failing, or some userland application going haywire, or even HW bug, I don't know. It was quite alarming nevertheless; with computers "anything can happen" even if it's not very usual.I'm sure Linux does better, but one can never completely rule out weird esoteric kernel bugs etc.

author: peter-h
author_rank: Super Contributor
timestamp: January 13, 2025, 08:26:45 am
content: Thanks5U4GB. I've passed that back.Fundamentally it must be difficult to make image backups of running systems, especially if the OS uses temporary files to carry interprocess messages. In Windows, whenever I make a Trueimage backup I reboot the PC afterwards, but that is a different issue to restoring such a backup. I currently have a problem elsewhere where a TI backup is not restoring to a running system so I will re-do that with a boot TI CD, to make sure the machine is not running.FWIW, the work which was being done on this machine prior to the image restore was setting up listing privileges on a particular directory, which I wanted to be invisible in a dir listing which currently is visible on a specific URL. That dir contains just one file which I wanted to keep hidden unless someone had the full path to it. I solved this easily by making it an aes256 encrypted zip. Normally this is done with htaccess but IIRC nginx uses a different system.That conf file needs to only contain 8.8.8.8 so that's been done and it has been made R/O. That seems to have solved it. Possibly some error log will appear in due course showing what may be trying to write to it

author: 5U4GB
author_rank: Super Contributor
timestamp: January 13, 2025, 08:47:03 am
content: As an example of sustemd's messing with things, from some old notes:Quotefollowed by something like a full A4 page of all the steps necessary to get it to stop breaking resolv.conf. One thing that comes up from those is to check whether resolv.conf is a symlink, in this case to ../run/resolvconf/resolv.conf, which means the problem is coming in from there.See alsothis link for another possible cause although that depends on use of udhcpc, including the malware-like behaviour where even the nuclear option of usingchattr +iwas bypassed.

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 13, 2025, 09:25:06 am
content: Quote from: DiTBho on January 12, 2025, 09:45:23 pmSure –– but those kinds of bugs can really produce all sorts of weird effects and problems!That is, the symptoms match only because of random chance, not because they correlated somehow.Quote from: Siwastaja on January 13, 2025, 07:53:32 amIt really does happen because of the different file system model in the Windows kernel. In the MS-DOS era, on FAT12/16/32 volumes, this was even more common. In both cases, it boils down to the file system model considering each allocated sector to be part of a file; whereas Unix/POSIX/Linux inode model combines file data and metadata (but not the name) into a logical object, that occupies a set of sectors.In the inode model, when a file is extended the kernel grabs some unused sectors, and writes the new or modified data to them, and finally the inode is updated to reflect the modified size and which sectors are allocated to this file. (Some filesystems also maintain a separate list or bitmap of unused sectors; this is usually updated last. POSIXfdatasync()returns when the data written to the file has been sent to the storage device, andfsync()returns when both the data and the inode updates have been sent to the storage device. In essence, this is integral to the inode model, and not just an implementation detail.The end result is that if the kernel crashes or power is lost in the middle of a filesystem metadata update, NTFS and FAT will generate unnamed files from the sectors that were allocated and saved but metadata not yet reflected in the file entry (MFT), whereas for inode-based filesystems such data is simply lost: the lost data was essentially written to unallocated sectors.I am not a filesystem specialist, and do not know exactly how much journaling helps (quantitatively, I mean). Anecdotally/qualitatively, it seems to significantly reduce the time window for the above to happen, making it much rarer, but cannot completely eliminate it.Now, the temptation would be to argue whether it is better to make the data recoverable instead of just discarding it. Fact is, most filesystemresearchin the last fifty years has used the inode model as a basis, because it has proven more efficient and more robust; the time windows during which crashes lose data are much shorter than they are in simplistic filesystems like NTFS and FAT/exFAT. (I'm not referring to Linux-originated ones like ext2/3/4 and btrfs, butXFSandZFS, and research papers on file systems at ACM, and presentations at USENIX.) To simplify, where FAT and NTFS were designed essentially for single-user workstation use, inode-model filesystems were designed to work for large multi-user concurrent systems where robustness is a key requirement.All this also means that writing or regularly appending or modifying important data files in a reliable manner differs between Windows and POSIXy OSes.Similar to how serial port/terminal-like devices are handled completely differently (POSIX usestermios), these differences make writing portable applications, services and libraries less desirable, because portability requires compromises affecting efficiency and robustness. (In particular, I've foundallcross-platform serial libraries' POSIX implementations pretty much utter crap. For data files, most developers don't even check their low-level calls for errors anyways ("I'll add them later", or "if that error happens, the user has bigger issues to worry about", and such inanities), so they mostly just throw data at the storage and hope it sticks, on all OSes.)My solution to this is to ignore Windows compatibility, and limit myself to POSIXy systems. It annoys me that others see this as dissing Microsoft, while it really is just about not willing to do the compromises required for compatibility between the completely different approaches. I'm not even saying the Windows approach is any worse, just incompatible with what I like to use unless compromises that I do not want are made! (Although, I am saying NTFS is simplistic and not very good compared to ext4-on-LVM, XFS, or ZFS, for the basic reasons outlined above.)Apologies for the rant-y post.

author: peter-h
author_rank: Super Contributor
timestamp: January 13, 2025, 09:07:49 pm
content: It was /etc/dhcp/dhclient-exit-hooks doing that .conf file "editing".We think it started when I restored a backup, but it is not clear why resolv.conf was empty after that. And that silly script just kept adding "options rotate".

author: Postal2
author_rank: Frequent Contributor
timestamp: January 14, 2025, 02:24:06 am
content: Quote from: Nominal Animal on January 13, 2025, 09:25:06 amI don't understand either, but I believe that the system will protect its files from failures, at least for Windows it is so. Then it turns out that the system does not consider the file resolv.conf to be its own, the owner of the file is someone else. And we come to the human factor again.

author: peter-h
author_rank: Super Contributor
timestamp: January 14, 2025, 07:01:36 am
content: Disregard this russian Postal2 troll and he might go away... Currently he is following me around the forum.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 14, 2025, 07:43:54 am
content: Quote from: peter-h on January 14, 2025, 07:01:36 amHe is not going anywhere. No idea why he was unbanned but we have to live with him. Of course you can add him to your ignore list but the saddest part is that he hooks beginners into his games by "helping" them, and wastes a lot of their time before they figure out what is going on.

author: Postal2
author_rank: Frequent Contributor
timestamp: January 14, 2025, 07:53:15 am
content: Quote from: peter-h on January 14, 2025, 07:01:36 amIt's a lie, I didn't even write anything in your thread about the toothbrush attack, although I really wanted to.Quote from: Siwastaja on January 14, 2025, 07:43:54 amDo I need to provide a link to the topic with the SPI connection problem, where you were the one chatting and misleading everyone?

author: peter-h
author_rank: Super Contributor
timestamp: January 14, 2025, 09:11:45 am
content: QuoteHe has set up some sort of "watch" on people and is following particular people around. Can you do a watch on username on EEVBLOG? Otherwise he reads every bit of the forum and watches out for usernames.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 14, 2025, 09:16:25 am
content: Quote from: peter-h on January 14, 2025, 09:11:45 amYou just probably feel that way. He just has a lot of free time in his hands, like many others. You give a lot of "reply surface" on your posts because you post a lot and in a style which encourages even a bit off-topicy replies, hence you get a lot of replies from all sorts of people, including trolls.

author: Postal2
author_rank: Frequent Contributor
timestamp: January 14, 2025, 09:20:11 am
content: peter-h, I don't follow at all and I don't read everything. But I know about your mania about being followed and attacked with toothbrushes.About "a lot of free time". I have 5 pdfs open on my computer, an audiobook playing, and I'm working hard (fixing a charger for a forklift). And, of course, I'm following peter-h.