Virtualising the home server. - Page 1
Forum section: electronics / general

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:02:50 am
content: So I got bored of the eWaste Dell Optiplex server, a disk failure also spurred me into acting.So I upgraded it to:* Asus B550 PrimePlus* Ryzen 5600G* 96Gb DDR4 3600* Dual Gen3 1Tb PCIe M.2 drives.* PCIe SATA controller card* PCIe 2.5Gb NIC* 4Tb SATA SSD* 2Tb SATA SSDThis then gets joined by the existing storage:2x6Tb HDD bulk media drives, no redundancy, small selective backups (stuff of my origin)3x batch matched 2Tb HDD backup drives (2 in RAID1, 1 spare)1Tb SSD + old 250Gb SSD (phasing out)Why do much RAM? Well, asides it being the number 1 route to performance I intend to use Proxmox and virtualize the whole host.The dual M.2 drives form a nice fast zfs mirrored pool of about 990Gb. With 1 disk failure recovery. 990Gb is more than enough room for half a dozen VMs/heavy containers with room to spare. "System disks" basically live here.The 3x2Tb HDDs could really switch to being a zfs RAIDz array giving 4Tb and 1 disk failure recovery. They can remain in the external USB bay and only be powered up for daily backups.I can't do much with one of the 6Tb drives, it's 80% full, so I'll just share it as is as a network drive. The other is about 20% and can be recycled to a different file system, creating a "zfs-slow-pool" with variably sized volumes.I'm still debating on what to do with the SSDs.On one end I have blisteringly fast 3500MB/s mirrored 1Tb PCIe drives. On the other I have huge slow, power hungry HDDs. Ideally the SSDs would go into one or two big cache arrays for performance. However, the easily available, tried and tested caching options such as L2ARC or even LVM-Cache just aren't designed for small servers. They usually get out performed by the host memory cache and never really taken advantage of unless you repeated read the same blocks of data.6Tb of SSD merged into one large volume could replace an entire 6Tb HDD. However that would involve a 2X multiplier on failure risk. If either disk fails, both fail.This has me considering making things a little more complex and possibly getting by with redundant arrays of "partitions" and maybe one more 2Tb drive.I have a 4Tb and a 2Tb. The best I can get out of that is one 2Tb mirror VDEV + a 2Tb single VDEV. Giving 2 pools, one mirrored one not.However a further 2Tb SSD allows me to create a 2Tb x 3 RAIDz array of SSDs which would be useful for the less replaceable bulk media, like the still not cleaned out "Camera memory card dumping folders by date, which is 0.6Tb now. If I spent a good while cleaning it out it would at least half in size, but... storage is cheap and I like to hoard. meh.M.2 = Proxmox boot/rootfs, VM boot disks, VM primary high performance data disks.SSD array(s) = Not easily replaced media/bulky items.HDDs = bulk media, no redundancy, limited backups, disk fail = data loss.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:16:41 am
content: Maybe... to keep things simple, I just don't employ the SSDs at all.If when the 1Tb of M.2 storage starts to get towards 50% or so I can bring the SSDs online and demote volumes from the M.2 to the SSDs, setting up the SSD filesystems for that purpose, when I have it.Having spare unused disks available can be very handy if/when you want to do some "lab work" with physical disks where VM disks just aren't going to cut it or you just want to do the lab with real hardware for whatever reason, performance testing say.

author: BradC
author_rank: Super Contributor
timestamp: August 25, 2023, 09:16:50 am
content: Quote from: paulca on August 25, 2023, 09:02:50 amI've been doing similar at home for years. The base machine now is a Ryzen 3950x with 64GB. OS/Home/VMs are on a RAID1 which is comprised of a 1TB NVME paired with a RAID0 of 4x500GB Samsung SSD drives which is marked "Write mostly". (board only has 1 NVME slot). This started life as a RAID10 of SAS 10K spinners, moved to a RAID10 of 6x250G SSDs (3 Samsung+3Intel) and finally to the current setup.The CCTV VM has direct access to 3x8TB WD Purples and a 6TB WD Red. The bulk of the storage is a RAID6 of 8 x 6TB WD RED. They're all > 6 years old now so I'm succession planning.The server is kind of a half/half. The base OS manages the VMs with Libvirt, file serves via NFS and SAMBA, and also runs the main mail service. Most other stuff is virtualised (CCTV, Zimbra, CAD workstation and so on). Probably have 8 VMs running now which are a mix from Windows XP through to Win10, and a couple of versions of Linux.Probably started doing it this way back in about 2006, and I'm on incarnation 3 of the hardware. It's easy to work with, versatile and means my desktop only has to be light-weight (it's still a Ryzen 5600G with 64G of RAM).It's a neat way to go. I look forward to reading more as you put it all together.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:27:26 am
content: Speaking of going way back...The new server is in "lab mode", the existing server is still valiantly serving. So the idea is to prototype it, give it a good hammering and then, possibly tonight, possibly all day tomorrow it gets wiped and the best of the best plans selected, implemented and the plan is to have complete server conversion by Saturday evening.I wanted to test ZFS arrays of partitions, so I went looking for old hard-disks. The gaming PC is propped up by a pair of 2Tb HDDs and a pair of 1Tb HDDs. But I need those to keep it up off the carpet as it has floor fans/vents. So I headed out to the garage and in amongst the dozens of dead spiders was the "old" stack of HDDs.The gaming PC is now propped up with a Maxtor 200Gb IDE drive, 2x320Gb IDE Hitachi Deskstars and a Maxtor 120Gb IDE drive, dated 2005!I didn't mount the older ones as I have no IDE interface, but the 2Tb and 1Tb SATA drives, two failed (both the 1Tb) the SMART test with age and wear. The 2Tb's are fine, one of them in fact was an offline backup drive, which if I can work out which it is from the SMART data should have very little "online time" on it.One more order coming from Amazon, finish work at 5pm, I wonder what problems I will face in migrating things and how little downtime I can achieve in the move.

author: BradC
author_rank: Super Contributor
timestamp: August 25, 2023, 10:40:19 am
content: Quote from: paulca on August 25, 2023, 09:27:26 amI rsync one server to another while the original is still live, then shut all services down and do another rsync to clean up.Generally when I do a migration I'll prototype and test it multiple times and build a script to do everything including diff/patch all the config files. Lots of reset-reapply cycles so when it goes for the final time all I have to do is run the script and reboot the new machine to bring it up.As for old drives, I still have a 12x2TB RAID6 here. Most of the drives have > 6 years run time on them and are pushing 12 years old, but it doesn't get much use now other than as a test mule so spends most of its time powered down.

author: PlainName
author_rank: Super Contributor
timestamp: August 25, 2023, 01:17:20 pm
content: I wen the virtual route a while back but did it mostly for the power saving (so picked low power processor rather than shit hot performer). However, I wound up with two hosts, a primary running the infrastructure stuff and NAS, and a secondary running not too important stuff (nextcloud, xmpp, etc).Reason for that is because having everything on one host is bad news if/when it dies - where do you start to get going again? You can't even do that unless you have another machine to either restore all your virtual machines, or dump a backup of the main server onto actual hardware. So the idea is that the secondary server has a copy of the primary virtual machines that are offline, and if the shit hits the fan then those can be started up to take over until the primary is sorted out (they are not backups as such so would be out of date, but they're good enough to keep all the plates in the air).And... also some microservers which are backup destinations. Again, I don't want backups on the host that's going to go tits up, and the microservers can be easily carted to somewhere else.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 06:30:52 pm
content: <waves>I was only offline for 5 minutes intermittently.* Recreate the existing server (temporarily) in a VM.* Copy over the "network-services" docker-compose stack.* Bring up the DHCP + DNS on the new server VM.* Shutdown DHCP + DNS on the old server.* Check. All good.* Swap the IPs.* All hell breaks loose as expected, my dhcp leases all expire and the network falls apart. Ubuntu "netplan" config, with a typo took me a while to find why I had no networking.Fix it though. So I have DHCP and DNS, so I have internet. Now of the rest of the services. Then the storage can move and the fileshares can come back up.I have until I have drunk 3 or 4 beers and then I have to stop.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:17:11 pm
content: Well, the important services are restored.5 hours. Most of those were wasted fighting with DNS. If it wasn't systemd-resolvd screwing around redirecting and caching DNS while I was debugging it was docker randomly choosing which 172 network it wanted to be on. Sounds easy, but figuring out why random parts of docker stacks won't come up took ages. Until I found the DENIED messages in the DNS server log.I still haven't moved any disks. The old server has been demoted off it's IP and it's aliases transfered.So far, so good.PlainName: I somehow feel very uncomfortable with the DHCP and DNS servers on a VM. When I reboot the supervisor host it just feels like a gamble. The supevisor host does not use the dhcp, but it does use the DNS in a VM... that it runs.I don't really want another new server, but instead I will add a TODO to make my DHCP/DNS a little more resilient. I have a two very powerful router/access points which run Linux (OpenWRT), I'm fairly sure I can get them to play nice as fall backs.... some how.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:20:10 pm
content: It's funny, but for me, a successfully running server includes things like:The lights work.The heating works.The internet works.Only twice did I need to turn the bathroom light on myself. How 1900s.

author: paulca
author_rank: Super Contributor
timestamp: August 25, 2023, 09:55:59 pm
content: On the power front.Jury is still out.Data for the last 24 hours shows....On the left the big flat portion around 45W is the old server "idle load" + the switches, routers etc in the office.Most of the bumps before "mid day" are probably the HDDs spinning up. They are "kept alive" between 9am and midnight to prevent constant sleep-wake cycling during the day.Just before midday the Amazon parcel arrives, the extra 64Gb of RAM goes in, my manager in work hand waves away me task for the rest of Friday.... so I fire up the new server. We jump 60W to 100W. 40W for the new server coorelates nicely with the WattMeter power plug on it.At 22:15, I shut down the old server. HDDs and all. Settling out at 70W.... for now.It's hard to tell. Worst case it looks like I'm 45-70W base idle load. There is some room for tweaking, but the HDDs are OFF and it's 70W. I might have work to do. 96Gb of RAM burns power.EDIT: Ooops. The other thing that arrived in the Amazon parcel was a 2.5Gb 5 port switch. That is going to add 5-10W to the equation and spoil the assumptions above. Again... I'll give the data a week or so and see.

author: BradC
author_rank: Super Contributor
timestamp: August 26, 2023, 01:20:13 am
content: Quote from: paulca on August 25, 2023, 09:17:11 pmIt's only a problem if your configuration isn't set up correctly. I have 3 VMs that need static IPs. They need static IPs because I have iptables rules that forward to them or filter from them and I can't guarantee the VMs are up and registered with DNS before the firewall runs. I simply have those 3 MAC addresses with static allocations in the DNS server. Everything else "gets what it gets", but as the names are properly registered it all "just works" (tm).The host server is running isc-dhcpd to serve addresses, and bind9 to manage the DNS. I also have a dnsmasq instance that can't resolve outside the network and I have dhcpd to push that to VMs by default. I needed that because part of the Zimbra server is using dns to exfiltrate data and license information. Like the static IP, if a VM actually needs to see the world, there's an exception in dhcpd.conf to push the main BIND9 instance as its DNS resolver.

author: PlainName
author_rank: Super Contributor
timestamp: August 26, 2023, 09:54:19 am
content: QuoteIt can be a problem after a lengthy power outage (that is, when the UPS for that server has expired). The host and VM take a while to get going; meanwhile lots of small stuff has popped up and given up waiting for an address... In that circumstance I just wait for everything to get powered then recycle the power (or other wise reboot) for everything except the DHCP server. Bit of a drag, but doesn't occur often.The advantage is if the hardware goes bad the VM can be run up on my PC if necessary. Or laptop, even, and we're off again.I periodically think about alternatives for DHCP (DNS not such an issue - doesn't matter if it takes longer to come up so long as it does eventually). The one I use has a particular feature set that I like so I'm reluctant to change, and the time issue is with the host booting, not the app. I could turf it off to a Pi or something, but then that's another thing to maintain and fix when it's bored.

author: paulca
author_rank: Super Contributor
timestamp: August 26, 2023, 11:29:49 am
content: I'm going to explore running ISC-DHCP on the OpenWRT router. Linksys WRT3200AC which should have enough to run isc and a package exists. Possibly just as a "failover peer" but maybe, if it supports it a hot/hot pair.However, I stopped myself just in time and put the backups first on the list. There are package updates on the main router. I am not touching that until I have a full recovery image of it from a working state. That would not be a fun evening re-configuring it. Hopefully I don't brick it... I do have a fail back route though, the advantage of putting your WAN on a VLAN is you can pick up the PPP connection anywhere.Just moved over the main storage. I have to say, "Thank You, Fractal designs." Their case layouts, even their cheap cases, like this "Core 2300" have so much attention to detail where it matters. For drives they have a panel that unscrews (toolless), on one side you can mount 3 2.5" SSDs and on the other you can mount 3 3.5" drives with rubber grommet mounts. Then the whole thing just thumb screws back in.My SSDs are all lying in the bottom of the case for now as I am awaiting the IcyDock 6 bay hot swap SSD 5.25" module. 6 SSDs in the space of a CDROM drive. Should be interesting.

author: paulca
author_rank: Super Contributor
timestamp: August 26, 2023, 07:59:08 pm
content: Job done. I now have a HA pair of DHCP serversBasically this.https://kb.isc.org/docs/aa-00502So primary is the main server VM in proxmox, the secondary is the Linksys WRT router. Sweat.I may have to rejig the configs around so that the main bulk of it is an include which can be synced between the pair by cronjob. The only bit that differs is the failover peer definitions.

author: paulca
author_rank: Super Contributor
timestamp: August 29, 2023, 09:51:46 am
content: Ooops.Guess who didn't check the guest network? Hmm. Yea. Found out this morning that by changing the DHCP server on the Linksys main router to be the fail-over for the LAN DHCP... I completely forgot that the Linksys (dnsmasq) DHCP is serving the guest network. Doh!So, "No Internet" and no IP on the work laptops at 9am.One was easy to fix, I just set static IPs, DNS and gateway. The other needs admin access to change the IP, gateway or DNS settings without the admin password. It will allow anything you give it over DHCP, but static settings don't work.It gets worse. The LAN and GUEST networks are completely layer 2 separated. Exactly so that things like DHCP and broadcast can not bridge between them!Shafted by my own security.The only option I had was to try adding the other subnet to both DHCP servers in the hope that the failover-peer would respond to the GUEST interface and assign IPs from it's pool. A few failed attempts later and finally I discovered that, yes, if the primary doesn't (or can't) answer, (in my case the primary has no interface for "guest" and doesn't really know it exists, ... the fail-over pair will answer. I think if it recieves it directly and the primary doesn't respond it will. Either that or it's smarter than it looks regarding interface to subnet authority and automatically assumes authority for the subnets it detects on the it's "listen interfaces" and have declarations.The GUEST works again, however, it has no fail over. The guest VLAN cannot broadcast on the LAN. I can get the router to forward packets on UDP 67 into the LAN DHCP server and the firewall might route the response, but I don't see how the LAN DHCP server is going to determine that address should be allocated from the GUEST pool and not the LAN pool. I mean I DID get it to assign a LAN IP to a Guest by forwarding the DHCP port through. Not useful and the router will not route LAN addresses from the GUEST VLAN. A long and ugly shot would be to use fixed hardware-address based DHCP static hosts.The other option which I am considering, is using 2 NICs on the server. One for LAN and one for GUEST. FW the hell out of Guest and only provide DHCP and DNS. Thus providing the LAN server a direct interface context on the GUEST VLAN.EDIT: If I get lucky the 1Gb spare NIC in the server will support VLAN awareness and I can make it a trunk port and get all the VLANs.

author: PlainName
author_rank: Super Contributor
timestamp: August 29, 2023, 10:19:28 am
content: Doesn't your DHCP have the facility to look at the clients user or vendor class to determine which pool should be used?

author: paulca
author_rank: Super Contributor
timestamp: August 29, 2023, 11:35:02 am
content: Possibly, probably. However, I can't see it providing more than just MAC assignment.ISC is a bit weird with it's scoping of where you declare things and where it puts them. Host declarations are global and apply regardless of subnet. If hw addr X is assigned a 10.0.0.0 address, then it will be given that address regardless of what interface or implied subnet it hails from.I am not sure of the scope of the "pool" directive. I thought it was subnet->pool(s) but it may be possible to name a pool and assign it in a host directive with vendor/class matching.The work laptops are a Lenovo Thinkpad X1 and an HP Elite book. I probably "could" vendor class them, but if I buy a second hand HP or Lenovo laptop (Im considering this), that will go bad.I shall also check what client matching is available, such as "where did the request originate", was it IP forwarded from somewhere else or did it arrive on a local interface broadcast, for example.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 29, 2023, 12:47:44 pm
content: Why not usednsmasqinstead of bind?Even though some consider ISC Bind "the nameserver to use", itssecurity historyshows it is utter crap when it comes to anything security-related, even after the full rewrite (bind 8 and earlier were much worse). While that ofdnsmasqisn't perfect, it is somewhat better. Plus, dnsmasq is much easier to configure correctly.

author: paulca
author_rank: Super Contributor
timestamp: August 30, 2023, 09:45:09 am
content: I considered dnsmasq. But I have ISC working now for years and years.At the moment it is handling the two guest laptops by static hw-addr assignment. I can fix it later.So far Proxmox and I have been getting along "Ok". It is a bit bullying and a bit nannying when it wants to be, but it doesn't try and stop you going into the console and fixing things it can't yourself. Most, not all then reflect back in the GUI. The round-trip integration between GUI and config files seems to be quite solid.There are annoyances, like it's insistence on choosing what type of data can go in what type of storage and what type of physical device can go into storage types.For example. I made a RAIDz pool of 3 discs as backup-zpool. In it I wanted my backups, but also the most commonly used boot ISOs and VM/CT templates. However, it will only let you store backups, ISOs and templates in a "local" directory. It won't let you store them in a zfs pool or volume. However, it will allow you use a local folder ... even if that local folder is just the mount point for backup-zpool. This means you have to have 2 storage items for a single disk, one as it's zfs managed volume and one as it's mount point.A bigger question is if a bucket of water decends on the server and I have to buy another one tomorrow. Assuming my backup drive array survives... "How do I restore it?"So a small project is going to be, firing up an old Optiplex connecting that backup drive and seeing if I can restore the cluster onto it. (It won't have the power to run more than one or two VM/CTs, but it will tell me if I can resotre the config, vms and disks. That should generate a bunch of "bugs" and missing backups which can be corrected. A second related small project is taking a similar Optiplex and bring it up as a second node. This node can be schedule WoL booted and have key VMs and CTs migrated/synced to it, then offlined. Again this will be tested such that if I switch the main server off. I can boot the backup and it will be able to cover critical services whlie I restore the main box. The difference in this case is that I'm going to try and use the PVE replication features... without going to the full "Quorum 3 node HA model"

author: paulca
author_rank: Super Contributor
timestamp: August 30, 2023, 10:01:11 am
content: Another project (set) this has spawned is moving and rearranging my "public" DNS. My domain was held with a basic cPanel style host. So while I had full control to edit my zone file, I only had one zone. Any attempt to break out of that zone with NSs for sub zones was blocked. The cPanel did not support/provide any DynDNS or Rndc api either.So on Saturday morning I moved my main domain to AWS Route53. Given the domain is active and my primary email account hangs off it, it had to be handled with care. Which I should have reminded myself about when I filled in the registra information to use an email address within that domain and even hand waved AWS away when it warned me about it. Thankfully the domain transferred without issue and I got all the emails from nominet uk, AWS and the previous register. Phew.While I was there I scored a nice short easy to remember/type domainname. This will be used for "project new lab domain". A completely isolated, from the ground up fully hosted domain. Hoping to make use of AWS SES for outbound emails. Possibly a pre-canned mail gateway solution as the front end relay and then a basic postfix underneath. There are several fully canned "Mail server in a box" appliances like mailCow and Zimbra but ... they are overkill, fiddly to setup and usually expect to send email directly with all the reputation feedback flows and server identity management etc. Zimbra looks like it's gone split model payware/oss which looks to be very agressively marketed.I ran my own raw SMTP based domain email for a long while before. It just isn't worth doing as a full top to bottom stack these days. Email has become a lot more secure with the big players like google, apple, microsoft enforcing them the most. So even just sending email to a gmail address these days requires a lot of hoops to get through to gaurantee delivery. Thus using the likes of SES which "lends you" some of AWS's trust to begin with and if you abuse it, you get dumped to the non-approved range which have been blacklisted already by others. On the receiving end, unless you are careful the sheer amount of spam and back scatter passing through your mail queues will overwhelm your disks and server. Previously when I gave up on *@domain addressing and limited it to a selected few, I made the mistake of "rejecting" unknown mailboxes. This ultimately was the decision that overloaded the server with mail relay reject message loops, back-scattering and also got my IP blacklisted. Not as a spam or open relay host, but as a badly configured backscatter host.So I'll start with the lab domain internal email, so that services, crons and daemons can spam me. Then I'll have that lab domain consider my LAN as the internet (with an admin NIC just in case). Hook it up to the internet via SES outbound and a mail gateway MTA for recieving (anti-spam, anti-phishing, black/white list, etc. etc.).The DNS reconfig allows me to have a publicly answerable ".lan." subdomain. While this (and other route53 functionality) has allowed me to register SSL certificates for my LAN and get rid of the security warning on HTTPS APIs.The difficult part is that I already run a split horizon DNS system. My local DNS is authorative for the ".lan." subdomain. Queries for the parent go outbound. Queries for LAN answer locally. I "could" move the whole lan subdomain to AWS as a secondary (or primary) and just put local addresses into the zone file, but I don't really see the point and it would expose local network details about all the routers, switches etc. So a stupid idea.So I need to work out if and how this can be useful. Possibly just setting the ".lan." NS to a local address instead might fix it. I only expose 1 or 2 local IP addresses publicly, but from my point of view lan. would still answer with local authority. Even if my actual DNS server is set to 8.8.8.8/8.8.4.4www.lan.domainwill resolve domain publicly, get the nameservers for "lan" as like 10.0.0.1 10.0.0.2 and then recurvively resolve LAN addresses from there. To behonest this sounds like a mess waiting to happen and doesn't provide me any functionality other than not having to configure a custom DNS server locally.I may just leave the public "LAN" record as an A record or a CNAME for whatever the current public IP is.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 30, 2023, 10:08:10 am
content: Quote from: paulca on August 30, 2023, 09:45:09 amThat is sufficient reason, in my opinion. These are only tools for getting things done, and they will never be perfect anyway.Quote from: paulca on August 30, 2023, 09:45:09 amYup.I've learned to ask myself every now and then, what would I lose if my toolXwere to give up the ghost right now. (As I've had that happen a few times along the decades.) It is a very good reminder and a yardstick on whether one wants to spend the extra couple of minutes to make sure one has an offline or remote backup of the Important Stuff. I surprisingly often find it worth the effort and time (to make an extra backup, or verify the current automated backup method is working as intended).

author: PlainName
author_rank: Super Contributor
timestamp: August 30, 2023, 11:03:23 am
content: QuoteThought a lot about this, and the lack of live backup in free ESXi was a factor...I figured the important stuff wasn't the server with all its VMs but the VM contents. So I treat each VM as a standalone machine and run backups on them as I would with a real PC. The idea is that a restore would be the same as any PC, just what it's being restored to matters. So if everything went tits up I would just grab the nearest host (real or virtual) and restore the backup on the disk (real or normal) as usual.The VM server is nothing special and could be whatever is flavour of the month. Create a blank VM with appropriate storage, restore the backup into it. Sure, it takes a but more effort than pressing a button and waiting 4 hours for the server to be restored, but I think it's a more robust solution.[The backup software I tend to use is DriveSnapshot, which is pretty ancient but low resource and Just Works.]

author: BradC
author_rank: Super Contributor
timestamp: August 30, 2023, 11:05:17 am
content: Quote from: paulca on August 30, 2023, 10:01:11 amIt's really not that hard. We've been running "email from home" since 2003 now. I have a cheap VPS in Sydney I use as the main outbound relay (because I can't delegate a reverse on my home VDSL) and have had zero deliverability issues to Google or Microsoft services for the last 20 years.All public facing MXs are Exim4, and they relay to and from other services (such as Zimbra internally for the business E-mail, which gives is MAPI for our last Outlook holdout and ActiveSync for all our mobile devices). When the inevitable downtime occurs, there are 2 secondary MX (the VPS outbound relay being one), and I have a backup outbound relay in the UK because when I started the VPS with Exigent their cost far exceeded their reliability (and they were cheap). Since they sold out I haven't had an issue, but retain the UK Co-Lo machine just in case. The UK stuff is a hangover from when I was based in the Middle East, and they were the shortest hop to reliability.Having a look at my dhcp setup, I have 3 pools. Authorised, Guest and VM. These all have different subnets, interfaces and dns instances on the same server. It took a while to get refined, but works pretty well now. I don't have a backup dhcpd because if that machine shits the bed the entire network is off-line anyway. I have a dev machine with identical hardware that holds a cold-backup of the basics should I need to bring it up and can restore last nights backup to that if required.Backups are nightly on all machines onto a single dedicated raspberry pi and a couple of USB drives. Currently I have on-line nightly rotating rsync backups that go back to Feb 2020 for every machine in the system. For the 3 years prior to that they're stored on a cold-drive in a safe. Nice when your cheap-arse remote VPS host shits the bed and you can just rsync last nights backup onto the new machine and be on-line in 20 minutes.Each backup snapshot file is stored with a sha256 of each altered file in that set, and every night the "next" machine is selected for a full "rsync -c" of the entire tree to catch any silent corruption on either the backup drive or the source machine. Every friday night, the backup machine traverses every backup in the tree and checks the sha256 of every file to detect issues with the backup. I also keep a sha256 of all the hashes, so they get checked also. I had an issued years ago with a Sil SATA controller and silent corruption. Once caught!I now also run monthly backups to tape which are stored off-site.Having dns spread across bind9 and 2 dnsmasq instances allows me control over what sees what. I found Zimbra exfiltrating data over DNS. None of the VMs have external network access, and all of them use a dnsmasq instance that only delegates specific internal domains (and CLAMAV) for resolution. The rest return null results and all are logged. I then get an E-mail once a day with the list of domains and machines that tried to access them. AutoCAD is particularly nasty for trying to phone home in many ways.If I need a VM to be able to access the outside world, I just add its MAC to the relevant DHCP pool and it gets assigned a NATable address and working DNS.Wow, that rambled. Sorry about that. Yes, doing mail at home is easy and once it's configured it's pretty set and forget. Just make sure you have a proper reverse, SPF, DKIM & DMARC.

author: paulca
author_rank: Super Contributor
timestamp: August 31, 2023, 09:48:10 am
content: Ooops no. 2.I have been splitting up my main docker-stack based on the types of service. Last night I decided to move the influxdb and grafana to a dedicated LXC container and put the DB data volume onto an expandable zfs volume.While I was in there I noticed a service in the docker called "mqtt", which had been moved to "network-services" stack several days ago... or so I thought ...I removed it. Confident that things where working without it.A lesson. In docker, DO NOT, name your services with the same name as hosts. If you use service based hostnames (like I do), then you are likely to have an "mqtt" host... which I do. If you also have a docker service called "mqtt", then dockers internal DNS will present it first. When you have 3 individual MQTT servers and you start moving them around and readdressing them, then checking all the services refer to the "mqtt" hostname, you may be fooled into thinking it's all working. Until you delete the "mqtt" docker service.What had apparently occured is that some of the services where defaulting to "mqtt" as the hostname and no using IPs. Well, they really should have been using "mqtt" but most were not, or I would have noticed sooner. The thing that was defaulting to "mqtt" turned out to be the master clock for the home_automation system. The same clock which generates the time sync, heartbeat and keep alive's for various components.It's summer or I would have noticed the lack of heating. With no clock it just won't work. The lights don't care about the clock, they worked.Fixed now and from now on I'm using suffixes or prefixes in docker naming. "mqtt-srv" rather than "mqtt" or "dockerMqtt" or "d_mqtt" so it's much more obvious. There may also be ways to change the scope of the docker dns entries, but I believe they will be defaulted to one DNS per subnet, so one docker network, one DNS scope.

author: paulca
author_rank: Super Contributor
timestamp: August 31, 2023, 10:18:02 am
content: BradC,Your mail setup is fairly impressive and I expect it takes a bit of maintenance. I shutdown my mail server and moved to GMail (Google GSuite, now workspace?) which was free at the time. Now I have to subscribe to at least one google service to be allowed to host a domain in a "free account". It's kinda weird, but I just upgraded to 50Gb of storage for £1.50 a month and apparently that was enough.The reason I shut it down was the sheer volume of "churn" I was facing. I was (stupidly) returning "No such mailbox" rejects. However, spammers where just using those to generate emails from my domains, because there was a trusted SMTP MTA MX listening on it, so it was a "more routable" sender address, even without a mailbox.So I was then receiving rejects from other mail hosts for spam delivered as coming from my domain.... which did, very often, not comply to the standards allowing MTAs to detect mailer deamon rejects and not reject those... a lot got rejected back. This ultimately ended up in 1.5Gb of churn in the mail queues and 20% CPU load... on good days. On bad days it peaked at over 4Gb and my poor Athlon 64 and HDDs couldn't cope. I did set about trying to fix it, but along the way it all stopped working and... investigates showed I was listed on several black-lists for badly configured backscatter host. Sending was very paritial and GMail etc just wouldn't send receive from my domains at all.I decided to give up. This was right around the time that the various reputation and MTA verification mechanisms started to surface, so I was going to have to step up my game to keep it working anyway.I can obviously see some of the mistakes today. I would do it differently. I do think going "second level" behind a cloud fronted mail gateway, like AWS's is the way to go on the send and receive. They can do all the complicated stuff, spam filtering etc. So I at least get a moderately clean feed. Also, SMTP rejects for anything other than a local client should be silent and "DROP".The DHCP on my router was dnsmasq and it was set to handle the GUEST, ignore the LAN as it had it's own DHCP. The router is an easy choice as it has interfaces on all VLANs. The server, is not VLAN aware yet and only exists on the LAN VLAN, so it is a bit more tricky to server multiple subnets.My DNS is "fine" for now. I just split horizon things and enforce DNS routes in some cases.The LAN DNS server is the primary DNS given out to all machines. It is authoritative for "lan." and all clients are told to search that domain first. So locally defined zones get priority. No recursion happens on this first scan of local zones. If you ping the parent public domain however, the local bind9 forwards it, doesn't recurse it, just forwards it to "PiHole on 5353"From the guest network, port 53 is force redirected to PiHole on 5353 (Masqueraded) and more heavily filtered. For example the NSFW domain blacklist in turned on in PiHole for the guest network.I'm just going to keep this setup, at least on the primary functional domain. On public resolvable hosts go to route53.

author: paulca
author_rank: Super Contributor
timestamp: August 31, 2023, 10:37:35 am
content: So far..."erigal" is the VM created to move the existing server too. It's rapidly shrinking and I think it only has two things remaining to move to unprivileged containers.The 96Gb of RAM definitely helps, running just these "24/7" service hosts the box isn't even taxed.It's not all perfect though. I repeatedly get very high I/O Wait states. I suspect proxmox may be doing there thing of enforcing stupidly high standards on integrety and security that disks are written to synchronously in many cases. Copying from SDD to a USB HDD as root for example will bind all cores into a I/O Wait state and case huge delays on the VMs. Its almost as if the core that initiates the write to the USB HDD has to literally sit and wait at 100% IOWait until it has been written and confirmed.... every block. This isn't acceptible and needs looking into. If a user (me?) wants to copy 2 Tb of data between any two drives the system should not go into a high load bottlenecked status!Some of the way proxmox "classifies" storage can get irritating. You end up with really janky stuff like the host mounting an HDD so that the NAS can access it and share it with NFS, so that the PVE nodes themselves can mount that storage again and use it as "storage". It's really confusing, but mostly stems from their storage classification as "ZFS, Directory, sub-volume or raw disk dataset." Each has it's own different set of properties and usability. The other aspect is that unpivelged containers cannot mount NFS shares. I mean, you 'can', but it's a hassle and even when you do you have to content with mapped UIDs and GIDs. Thus you end up relaying FS's through local mounts via virtual NFS shares and the reverse. Just to share filesystems. As far as proxmox is concerned, filesystems should never be shared. It tried to force you done this path at all points and there are literally thousands of posts a year from people confused as hell about how to simply share storage. Stupidly, dangerous, advanced and complicated approaches include... installing a dedicated SATA controller and using PCIe passthrough and IOMMU to "give" that controller entirely to the VM.

author: PlainName
author_rank: Super Contributor
timestamp: August 31, 2023, 11:41:44 am
content: QuoteThat kind of thing is what puts me off installing some otherwise useful stuff. Once it's in place I'm not expecting to become an expert and maintain it all my working hours, so the chances are that when it goes wrong and I need to fix it I'll be up shit creek without even a google search to help. BTDT, prefer something simpler and obvious.

author: paulca
author_rank: Super Contributor
timestamp: August 31, 2023, 12:21:18 pm
content: Quote from: PlainName on August 31, 2023, 11:41:44 amYea. I believe it comes down to Proxmox wanting to maintain it's dynamic provisioning so it can support things like replication and HA of VMs and disks. If you want it to be able to migrate a disk or filesystem it kinda has to be one of "it's" managed filesystems.However. As it is very unlikely I will be using Proxmox HA in the foreseeable future and if I do "replicate" to an "cold standby" host, I can facilitate that myself, if needs be.So, the "PVE" node is the master node. It's the only node. It also, physically holds the disks. Thus, in my view all of these disks should be, across the board, mounted normally on the PVE node root FS and available to all valid users.That single folder, say /srv/ or just /disks/ can be mapped to the NAS container and served as a series of NFS shares.All VMs can mount said NFS shares. (SMB/CIFS for windows)All containers can map which even part of the /srv/ folder they choose.Any disk/filesystem I fore see the need for proxmox's management help, I can provide it the right config to manage them... or let it create those disks.I'm starting in the CLI. Using only ZFS and normal tools to get all the sub-volumes and drives mapped into /srv/. It would appear that while I am doing this Proxmox doesn't seem to care. It shows new ZFS pools and volumes I add, but otherwise leaves me alone.tldr; it can be simplified in the majority of cases and I only need to expose myself to Proxmox management if it will be to my advantage.

author: paulca
author_rank: Super Contributor
timestamp: August 31, 2023, 02:27:46 pm
content: Quote from: Nominal Animal on August 30, 2023, 10:08:10 amHmmm. Then I seen this...Quotehttps://www.isc.org/dhcp/- Access date of post.

author: BradC
author_rank: Super Contributor
timestamp: September 01, 2023, 12:14:51 am
content: Quote from: paulca on August 31, 2023, 10:18:02 amMaybe an hour a year for security updates. Mostly an "apt-get update". It just runs. Because there are secondaries and the DNS is redundant, when a remote service goes down in a maintenance window failover is transparent, as is fail back. I get an E-mail to say "xx is down", then some time later "xx is up".I'm lazy, if it takes ongoing effort to maintain it's a non-starter. There's always a bit/lot of up-front configuration, but once it's running it keeps running. I'm very much a disciple of "as complex as necessary, but as simple as possible". I don't have time to have to fettle stuff to keep it running.

author: madires
author_rank: Super Contributor
timestamp: September 01, 2023, 12:00:54 pm
content: You can find several webinars on migrating ISC DHCP to KEA athttps://www.isc.org/presentations/.There's also a configuration conversion tool available.

author: madires
author_rank: Super Contributor
timestamp: September 01, 2023, 12:33:42 pm
content: Quote from: BradC on September 01, 2023, 12:14:51 amIMHO, it's hard to stay lazy as everything is developing further. SSL/TLS, SPF, DKIM, DMARC? All the scanning by the bad guys? Two months back there was a new SPAM compaign using RCPT flooding (never seen that before). So we had to create a new filter for the MTA config to detect the RCPT flooding attempt and limit its impact while adhering to the email RFCs.

author: BradC
author_rank: Super Contributor
timestamp: September 01, 2023, 02:49:18 pm
content: Quote from: madires on September 01, 2023, 12:33:42 pmNah, SSL/TLS is on by default on any sane mail server, just replace the keys with verifiable keys. SPF, DKIM & DMARC? I spent 2 hours in 2018 setting it up, 30 minutes in 2019 setting up automated key rotation. Haven't touched it since.The biggest thing here is ensuring the mail server can look up routeable e-mail addresses so others can be rejected in the initial handshake. Because we're a home/small business each domain (currently hosting 16) has a simple file of e-mail addresses. A watcher monitors any changes to this and propagates to the secondaries so we never send bounces from any domain or MX. Bots give up pretty quickly.That'd be more complex if the mail system was larger, but then I'd just automate it a bit more. Currently adding a user is a line in a text file. Even adding a new domain is just a text file. If it's local it's a list of address. If it's acting as a smarthost then that file also needs a destination MX address (for example we were running a VM with Exchange/BES for a while alongside the Zimbra VM). I really hate exchange, so I popped a cork when the last user migrated away from the Blackberries.It just doesn't have to be complex or difficult. I suppose the trick is knowing the traps, but I've been running company mailservers since Fidonet and things haven't really evolved that much.

author: paulca
author_rank: Super Contributor
timestamp: September 01, 2023, 06:14:38 pm
content: Test outs."Work" style self learning labs.* Kafka + Zookeeper + KafkaConnect w/ MQTT connector - Single unpriv'd container running nested docker stack.* Windows 10 Pro VM with 2 x IntelliJ and Java projects, local docker (via WSL nested HV on host's type 1 HV CPU).* GitLabCE full stack CI/CD stack - very nested docker and docker within docker + native servicesWorks. Shutdown down as easy as it starts up. Uses about 30Gb of RAM and the Windows VM generates up to 8.0 load. You notice it's a single "consumer" grade hardware node, but... you accept that and you ARE a single user.Today's test was:* Kali Linux VM.works.* Move it to a new bridge on it's own.Works, no network.* Add a pfSense VM onto that bridge with a WAN interface on the "real world lan".Worked. With a little work Kali got DHCP from pfsense and internet worked.* Enabled Squid proxy w/ cache on pfSenseworked.* Force the HTTP proxy transparentworked.* Force turnnel HTTPS through proxy transparent....FAILED. Seems pfSense (arguably correctly) suggests it's HTTP only or HTTP+HTTPS+MITM full SSL "filtering".EDIT: That's basically like, 2 labs and a set of core services and I'm still only showing 25% ZFS usage on the 1Tb M.2 local FS. I have only hit 90% used RAM once, but that 90% includes a lot of "cache", it's just how Proxmox tries to give you a real picture of whats going on, as it's possible to over allocate ram and storage.I have yet to try (and not right now!) to run all VMs and containers at once.

author: paulca
author_rank: Super Contributor
timestamp: September 01, 2023, 06:26:00 pm
content: Security awareness is important while messing around with these things at this level.For examples. Tonight I was trying to configure a trunk link to the second link on the server. That worked fine. Expect the server has ipv4_forward enabled. So the instant it had an VLAN interface on the guest VLAN and lan VLAN it *routed* them. Not ideal.I have no experience of "NIC" with VLAN. For me all 'access ports' are normally untagged and thus the VLAN is enforced.It would seem if you bring up 2 VLANs on a Linux bridge... it will bridge the VLANs. Also, if you also have alternative switch routes that bridge will create a switching loops (without STP protections).How I managed to learn all that and not for a moment be offline of even loose access to the PVE node and VMs a testament to virtualisation.It did mean for a while the work laptops and guest network had access to the PVE node and it's web UI!There is a VLAN I switch on the main trunks. VLAN 101. It is the WAN network. Literally it's the bare ISP VLAN. Thankfully accidentally bridging it is impossible as it's brdiged before PPP/CHAP auth. I "can" route it to any node on the network to PPP/CHAP it and present the public IP, but it can't be accidentally bridged.

author: paulca
author_rank: Super Contributor
timestamp: September 01, 2023, 06:36:49 pm
content: To solve the NAS mounting confusions.I mount all the local filesystems and ZFS volumes in /srv on the physical node. Then a full priv'd container imports the mount points (using a script) and exports it all using a single NFS crossmt export and a SMB share from the NAS /srv . (Basically a set of bind mounts -> NFS crossmnt export + SMB).For Proxmox disks and volumes... they can stay on the 1Tb M.2 storage for now.

author: madires
author_rank: Super Contributor
timestamp: September 02, 2023, 10:37:47 am
content: Quote from: BradC on September 01, 2023, 02:49:18 pmYep, automation is important. But I'm a bit wary of the configuration frameworks the distributions come with. Adding special setups to the framework can be a hassle. Also, I prefer to compile a daemon just with the features needed, for security reasons. It takes more effort but you lower the attack surface.Quote from: BradC on September 01, 2023, 02:49:18 pmI totally agree. Textfiles rock and Exchange is a nightmare. I had the pleasure to rescue a few Exchange servers in the past.Quote from: BradC on September 01, 2023, 02:49:18 pmIt depends on what you're doing and what you need. I don't do crazy stuff but often need some special features/services. For example, the latest exim version broke the UUCP transport by changing some internal variables to a 'tainted' state. They won't let me become lazy.BTW, I've written some FTS software and was member of the FTSC.

author: paulca
author_rank: Super Contributor
timestamp: September 06, 2023, 07:41:45 am
content: Well, I took the plunge and started a local mail server again. This time it will be split into two. Local mail server + Mail gateway. The reason is simple, the anti-spam/anti-virus and various filtering can all go on the gateway, which can then be firewalled to the hilt and port 25 forwarded in. I may move it to it's own network.All working so far, though I have not yet put Spamassasin or ClamAV in the gateway yet.Inbound works fine. Outbound, not yet. Awaiting "prod access" approval on the AWS smart host service.Also, my main email is staying with GMail for the foreseeable. Mostly this account will be for "lab work". 90% local daemon emails and hopefully very little else.It is nice, when an internal application asks for your email address and you just put paul@lan or even (though I have not set it up yet), just "paul". The later might require client side config.