Work Rant - Page 1
Forum section: electronics / general

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 06, 2016, 06:40:36 pm
content: So today I went into work as usual (I work at a CNC machining plant that has 30 odd BIG CNC centres running all the time: some of these are as big as a small house!)We had a powercut last night - the power was out for two hours... For some reason the backup generator didn't kick in. That was problem one. Problem two was that the UPS in each of the two automated vending machines died when they started trying to give a load to the computer, the monitor and the vending machine tied to it. Problem three was that when the power came back on, the UPS in one of the machines decided to let out the magic smoke. Problem four was that because the UPS was a little dicky that it corrupted the databases on the tools in the vending machines... Problem five was that the database backups were stored on a USB pen drive that was also corrupt, problem six was that nobody could get tools out the vending machines so they couldnt do the job... problem seven was..........................................................It seriously went on like this for the entire day! Problem after problem after problem - all because the power went out for two hours in the early morning!And the best part was that ALMOST EVERYTHING was preventable!The backup generator didn't turn on because nobody thought to maintain it (you know, just in case the power went down) and I'm not even sure it had petrol in it.None of the UPS had been checked since they were installed in 2008 - we had no way to bypass the UPS properly because nobody had thought they might fail when they were being installed - a working UPS would have prevented the database getting corrupted - the backups shouldn't have been stored locally (what idiot sets up automatic backups to run off a f***ing USB pen drive?We have a network with dedicated backup servers for f**** sake!)ANYWAY -Rant over!Anybody else had a bad day at work? Feel free to rant awayCorporate incompetence welcome!

author: ez24
author_rank: Super Contributor
timestamp: May 06, 2016, 07:14:12 pm
content: Just curious - what country did this occur in ?

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 06, 2016, 07:20:01 pm
content: Scotland (UK) xDThey've since rectified the problems - but jeez! What a day - FAR too exciting for a Friday!No other place I've worked at has been that badly disorganised when it comes to backup power. It's just fallen by the wayside from lack of use =/

author: kripton2035
author_rank: Super Contributor
timestamp: May 06, 2016, 07:20:17 pm
content: usb pen drives are for data moving between computers only ...surely never use them for backup they are no trustable at all.

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 06, 2016, 07:33:12 pm
content: QuoteI almost screamed when I figured out that's what they were using it for.It was actually a consultant who set that up before I even joined the company. That selfsame consultant was also a fan of writing his own applications but not documenting them; you can guess how much time has been wasted fixing messes that should never have existed.Thanks shambling corporate presence!

author: Wilksey
author_rank: Super Contributor
timestamp: May 06, 2016, 08:10:32 pm
content: Corporate Incompetence...or incompetence in general is all far too familiar for me also unfortunately! And I suspect quite a number of people.You know what they say, if you can't beat them...Imagine beating them with a very large heavy object!!

author: Tomorokoshi
author_rank: Super Contributor
timestamp: May 06, 2016, 08:36:21 pm
content: Quote from: Wilksey on May 06, 2016, 08:10:32 pmLike a purchase requisition, perhaps.

author: Kilrah
author_rank: Unknown
timestamp: May 06, 2016, 08:39:57 pm
content: This placethat happens to be the other tab in the window I keep this forum open in is made just for those stories!

author: sokoloff
author_rank: Super Contributor
timestamp: May 06, 2016, 08:41:31 pm
content: I ran quite a few small datacenters in my prior job. After getting my fingers burned a couple times, I made it mandatory to do site-acceptance tests on generators, ATS, and UPSs, and to repeat those tests quarterly in "test mode" and annually in "real life mode" (meaning actually cutting the street power). In 8 initial installations, I can't recall a single one that didn't reveal problems during initial commissioning testing.We also had ATS/generator controllers that exercised the generators weekly, bi-weekly, or monthly (depending on the DC and the local ordinances)Whatever form of backup you're talking about, if you didn't test it, you might as well not have it.

author: ConKbot
author_rank: Super Contributor
timestamp: May 07, 2016, 01:59:00 am
content: I worked at a cnc shop, loading machines, my first job. The business park the shop was in was in an area near 2 major portions of the grid, so the owner had a dual grid supply to the shop. If one failed the building would switch over too the second grid, switch back to the first when problems were had on the second. So unless something huge happened, the shop would always have power. (maybe the utility could command it to switch for load balancing too? Balancing for the utility, increased uptime for the business owner?)The problem was switching would take 2-3 seconds, so the power goes out, the machines shut off, the metal halide lights go out, and we're running around in the dark shutting off power to machines until the spindles stopped before we turned them back on. (50kw, 20000rpm spindle can turn for a while.) Not sure why, guess they had problems in the past. But it was always a mess when it happened. Usually all the tools other than taps were fine though.

author: Alex Eisenhut
author_rank: Super Contributor
timestamp: May 07, 2016, 04:37:37 am
content: Quote from: cprobertson1 on May 06, 2016, 07:20:01 pmGasp! Please tell me the distilleries are OK?

author: crazyjim30
author_rank: Contributor
timestamp: May 07, 2016, 04:49:14 am
content: Several years ago, I worked for a company that had several outside contractors with a long term presence on-site. The tank farm designated for contractor use was over a mile from the shop area. The 500 gallon tank for the backup generator... wasn't. They stated siphoning gas from the tank for convenience. This worked...until the power cut out and the generator tried to run on an empty tank. People lost jobs over that one.Sent from my SM-G900V using Tapatalk

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 07, 2016, 12:02:14 pm
content: @~ Alex Esenhut: No worries! The distilleries are fine! Do not panic!@ConKbot: Aye, those spindles are BIG - and if the turning machines have a job in the chuck that's a lot of inertia to dissipate.That' some massive inductive loads from those motors as well: I hope they were sensible enough to open all the circuit breakers on the machines before the power came back on!@sokoloff: I'm going to have to emphasise how important backup is; they'll say "oh it's not a problem, we'll just replace the broken stuff" of course and then not maintain it properly again.I've discovered that management don't like learning experiences.@Tomorokoshi: nah, you can't beat anybody with a purchase requisition unless you get a purchase requisition impromptu weapon license which needs to be sent in triplicate to your supervisor, charge-hand, HR and H&S departments before the requisitionforyour purchase requisition can even be signed off - by which point you've lost the urge to do anything let alone beat somebody with it!

author: switcher
author_rank: Regular Contributor
timestamp: May 07, 2016, 12:24:02 pm
content: Quote from: cprobertson1 on May 07, 2016, 12:02:14 pmAnd not forgetting that any purchase of more than five quid requires the signatures of three directors, at least one of whom is always on holiday in Dubai, and then it has to have final sign off by finance; who'll say " what do you want one of those for?"

author: MrSlack
author_rank: Frequent Contributor
timestamp: May 07, 2016, 12:36:39 pm
content: This sort of shit happens all the time even when well planned.Back in 2005 I worked for a company with their own datacentre with redundant everything and there was a supply surge from the grid due to a fucked up switch over. It took out the main distribution point instantly and permanently. No big deal, the backup grid connection will switch in instantly. It did and blew up instantly. No big deal, the battery backup array will switch in until the generators start, which it did and wow everything fine.Then the two redundant generators kicked in. Well one did - one of them cranked hard for 2 minutes and flattened it's own battery and failed. This left one generator online which promptly was overloaded due to a capacity planning failure and shut down after 6 minutes.And the guy who knew how to power sequence everything was somewhere in Brazil on holiday.Edit: Then our phone system in the office basically fell over from the thousands of people calling us wondering what the jell happened.That's also why you should always carry cash if that's a hint as to who it was.

author: uncle_bob
author_rank: Unknown
timestamp: May 07, 2016, 12:55:00 pm
content: HiI first ran into the "who needs to check the generators" issue back in 1964. Oddly enough this was in a "safety of life" setting (a hospital). The reason I was able to observe the whole mess was due to being in traction for a pair of broken legs and various other issues. So the life that the safety applied to was one that happens to be a bit signifiant to me.The drill apparently had been to check the generators on the state mandated schedule, on first shift. Oddly enough the lights went out on second shift. This being an era when automatic this and that were not as common as they are today .... nobody knew which switches to throw or levers to pull. The only guy who *did* know was an hour away and stuck in traffic (the whole northeastern US was blacked out). There being no cell phones, no way to get in touch with him. Bottom line, the lights stayed out for quite a while. All this was at a *major* hospital in downtown Boston MA.After this experience I sort of made it a hobby to look into the backup power systems at various places I found myself. My observation was that a *lot* of places did pretty much the same thing. Batteries were pulled out of the generators after "the test" and put back on a trickle charge on a bench. If you wanted to run the generator, you had to know how to put the batteries back in. All sorts or strange stuff like that was being done ...(Yes, I also tried to avoid ramming fast moving cars head on with my bicycle ... )Bob

author: SeanB
author_rank: Super Contributor
timestamp: May 07, 2016, 01:11:53 pm
content: Quote from: ConKbot on May 07, 2016, 01:59:00 amYou do get highbay lights with an integrated 500W halogen lamp added to the fixture, which is designed in with a current controlled relay, so that the light is lit when lamp current is low ( during strike, and when it is cooling from supply interruption), giving you some light when there is none. Works best with a mercury lamp though, as there is no complexity of a high voltage strike triggering the lamp.Only requirement is that you do actually change the lamps before they go fully EOL, as otherwise the continuous cycling will blow the halogen lamps prematurely, as they are only rated for 1000 hours of operation, unlike the MH which should do 15000 hours for a quality lamp, or the original GE/Sylvania/Osram/Phillips mercury lamps which could do 30000 hours to a 5% failure rate, and with the outliers going over 60000 hours.

author: radar_macgyver
author_rank: Frequent Contributor
timestamp: May 07, 2016, 05:34:02 pm
content: After a near disaster at my place of work, I mandated *weekly* backup system tests. We have a large air-inflated radome held up by blowers. There are two backup generators, one for the entire site, and one for just the blowers. When the power failed, the main generator did not come up because its batteries were discharged. The second backup came up, but, amazingly, the logic to switch to the second backup relied on either line power or the first backup being available! The person responsible for that system no longer works there. So now, we turn off the main breaker once a week to exercise the system. Once a month or so, we also turn off the first backup generator during the test to verify that the second one comes up.Most automatic generator start systems offer a periodic automatic test capability. These will not switch the load to generator, but will ensure that the generators starts and runs on some schedule (usually weekly). This will keep the batteries charged, oil flowing, etc. Same thing with UPS systems - most larger ones have options to add a monitoring card that will send out periodic emails.

author: Neilm
author_rank: Super Contributor
timestamp: May 07, 2016, 06:12:58 pm
content: I was at work about a month ago when all of a sudden I lost connection to the network. It didn't come back after a minute or so, so I tried to call IT. The phone was not working.Turned out that after they moved the IT server to its new home, they didn't bother moving the UPS that was too old and knackered. After the move, the powers that be decided that IT didn't need a UPS so canceled the order. Fast forward a year and maintenance are trying to work out what some of the circuits are in the switch room (the place dates back to the sixties and there are loads of cable runs that no one knows do what).So, giving this unknown they decide to switch the unknown circuits off and see what loses power. Sales office (no real bother) check, meeting room check, new IT server room (with all the servers including the phone system) oops.Major problems - all the machines had to be brought up and checked for issues. All work in the factory stopped for about an hour as the databases were checked and repaired. On a related note, IT is now getting the UPS they have been after.

author: stmdude
author_rank: Frequent Contributor
timestamp: May 07, 2016, 06:23:08 pm
content: Quote from: sokoloff on May 06, 2016, 08:41:31 pmPreach it! My favorite is the people having all the servers connected to a proper (and functioning) battery UPS, but leaves out the networking gear (since all the PCs will be powered-off anyways).Cue power-outage. UPS kicks in, servers are being well fed. All nice and dandy, right? Wrong. The UPS have no way of telling the servers that the battery is running low, and its time to power down gracefully, since the switches are powered down...

author: dmills
author_rank: Super Contributor
timestamp: May 07, 2016, 06:43:15 pm
content: Also fun when the NFS server is on the wrong side of one of those switches.Probably the finest power cockup I have encountered was a site with the generators on the roof (Hey, London property prices!), but the fuel tanks in the basement. Yes, actually it is somewhat important that the diesel lift pumps be connected to the redundant power main and not just the grid....Thing is they were doing hourly runs once a month and 3 hour runs once every few months, BUT WERE NOT TAKING THE MAIN POWER DOWN when running the tests, so they all passed.Regards, Dan.

author: GreyWoolfe
author_rank: Unknown
timestamp: May 07, 2016, 09:32:58 pm
content: Quote from: cprobertson1 on May 07, 2016, 12:02:14 pmAnd in my company, it usually takes 2-3 weeks to get the PR approved after going through the submission process so by then, you have forgotten who you wanted to beat with it and why.

author: sarepairman2
author_rank: Frequent Contributor
timestamp: May 08, 2016, 06:58:14 am
content: lol at my work place they would make the employees buy their own candles if power went out, then the safety guy would eventually make them buy flash lights and confiscate the candleseventually a generator would be brought in but you would need a manager to sign a use request to turn it on for a specific period of time. this would involve getting at least 2 signatures.there would be claims that the process is fair, however, in reality, when you went to the break room someone would "encourage" you to feel around instead of running the generator.what you need all that energy for? be reasonable

author: jitter
author_rank: Frequent Contributor
timestamp: May 08, 2016, 07:55:33 am
content: Quote from: cprobertson1 on May 07, 2016, 12:02:14 pmYeah, you don't have to tell me about that.Luckily data backup has been implemented correctly, but most other maintenance related issues have not. When confronted with this by the people who have to work with the machines etc. they just cite economic circumstances as the reason. And when something goes horribly wrong they take a few decisions showing they're clearly unaware of what's going on essentially leaving us to sort out the mess.Really, mismanagement is by far the biggest threat to the company I'm working for at the moment. But finally there are signs of the bigger shareholders becoming aware of this, so who knows what the future brings...

author: daqq
author_rank: Super Contributor
timestamp: May 08, 2016, 12:52:32 pm
content: QuoteI actually cringed when I read that one...For some reason I am the go to fellow when anything computer related happens to my family and friends - I design electronics, but in their minds it's still "computer guy"... and every year I have at least one person come to me with "David, my computer won't start and I need my photos, fix it please!". After finding a corrupt or dead disk I recover as much as I can, give a short lecture on the basic concept of back up (which will be ignored) and leave. Then I post a "public service announcement" on facebook urging everyone to make a back up on a physically different device, preferably more than one etc. One of them mentioned that he had two copies of all important files - on the same physical drive but in different folders.I swear that people do not understand the concept of a physical disk and they view data as some kind of esoteric, non-physical concept, a magic entity that travels with them... they do not understand that the data is actually physically stored somewhere, and destroying that somewhere means that the data goes as well. I'm pretty sure that there are people out there, who, if before their very eyes I took their notebook with all of their photos, music, misc. data and put it in an industrial shredder*, forced them to watch the whole process and gave them a new, fresh from the box notebook, started it up and put it in front of them, the first question would be "Where are my photos?".* -End of rant.

author: SeanB
author_rank: Super Contributor
timestamp: May 08, 2016, 01:42:31 pm
content: Only thing that that will work with is a Chromebook with an internet connection. Some seem to think "Cloud" is magic, and not that all it is is a return to the old days of dumb terminals and a mainframe, just with colour and sound. Otherwise they work in an office, where they either use a thin client, or have a computer with sync turned on to a server. Pop in a brand new computer ( with the right windows and office site keys,), set up printers if needed and then restart with the user credentials and ( eventually) the data is back. Leave the background as default Win7 and they might not even notice any change.

author: Kilrah
author_rank: Unknown
timestamp: May 08, 2016, 02:41:33 pm
content: Quote from: SeanB on May 08, 2016, 01:42:31 pmDon't be fooled, they'll certainly be moaning about that app shortcut that isn't placed at the same location than before with single-pixel precision...

author: SeanB
author_rank: Super Contributor
timestamp: May 08, 2016, 03:02:11 pm
content: Quote from: Kilrah on May 08, 2016, 02:41:33 pmLeave auto arrange icons on.............

author: AntiProtonBoy
author_rank: Frequent Contributor
timestamp: May 08, 2016, 03:35:01 pm
content: Quote from: daqq on May 08, 2016, 12:52:32 pmAh yes, that old chestnut. I'm the token computer person as well, or rather, used to be. The trick is to say flat-out "no" to any computer repair requests. I do this kind of shit for a living, as a part time IT admin, and I also work as a contract programmer. Basically spend a great deal of time in front of the computer, because that's how I earn my bread. The last thing I want to do is spend my own free time in front of other computers, dealing other people's crap for nothing in return.When holidays came along, such as Christmas, people pulled their broken laptops out, and requested me to fix their crap. Basically, sit in the corner and spend hours salvaging their gear, on my day off, on a public holiday, for nothing, meanwhile they were enjoying their time, socialising with everyone else. Yeah right, fuck that for a joke. So I flatly refused, only to get frowny faces in return and a one-way ticket on guilt train for not helping them out. But I couldn't care less at that point.

author: MrSlack
author_rank: Frequent Contributor
timestamp: May 08, 2016, 03:41:28 pm
content: I did that for years until one fine Christmas I was at the party clearing up some porn and spyware infested shitbox that my wife's uncle dropped on my lap. Took three hours.I sat and thought about it, then the day after I sent him an invoice for £120 for the thtee hours work.He didn't pay up, so I small claims'ed him and he paid up.Now no one every asks me to repair anything and I'm no longer invited to the parties there now.Apart from the woman in the local coffee shop that is, who swaps just the labour (parts she pays for), for an unlimited supply of breakfasts, cake and coffee. I'm happy with that deal

author: Kilrah
author_rank: Unknown
timestamp: May 08, 2016, 03:42:11 pm
content: I'm the opposite... pretty much hate those family meets on the usual holidays but obviously have to be present, so I'm rather happy if someone has an issue with $random_tech that allows me to go waste an hour or two away in a room while everybody else does the meaningless and boring random chitchat about the weather, how things change, the politics of the day etc as people who don't have many common things to share about do...EDIT: The thing that leads not to be invited to the parties anymore is something I should try again, but the previous attempt didn't succeed

author: Iwanushka
author_rank: Frequent Contributor
timestamp: May 08, 2016, 05:01:54 pm
content: I had the same issue with fix my pc/server constantly until one day I said fine 80eur/hour and I don't care if your dead pc takes 5h to install office, I will be sitting with my legs ontop of the table smoking and you still have to pay for those 5hours...Edit: average sallary is 400e, usual price for fixing pc was 7-15eur

author: daqq
author_rank: Super Contributor
timestamp: May 08, 2016, 05:03:17 pm
content: I see this is a common ailment on this forum... This is a very nice article on that topic, I'll probably start sending it to people:http://www.cracked.com/blog/6-reasons-guy-whos-fixing-your-computer-hates-you/

author: Howardlong
author_rank: Super Contributor
timestamp: May 08, 2016, 05:25:46 pm
content: Back in the mid 90s in a well-known investment bank when we started getting rack mount server hardware with redundant PSUs instead of towers, the tech (a guy basically making up the numbers waiting for the right severence package to turn up) plugged the two PSUs into the same power distro unit. Needless to say, it was only a matter of time.Earlier than that, in the early 90s, IBM were making a big sell on RAID. Problem was that the config was held on the HBA card, so when the card went la-la so did all your data. Then there were the IBM techs who came round with a small hammer to "tap" all the drives before powering up machines after a power interruption, because several heads would stick after the power outage rendering your entore RAID array useless. In those days, RAID was probably less reliable than non-RAID.But yes, we had two instances back in those days of the generator failing. One was because no-one had ordered any diesel, fella got sacked. Second one a few months later, no-one knew how to operate it. We managed to keep the machines going for half an hour on UPS, and did a controlled shutdown. Then we hailed a small convoy of cabs and drove the lot (about two dozen machines) across the Thames to our DR site, and had the lot up in under two hours operating a full production service. Most of that time was waiting for the CTO to allow us to physically take the machines out of the building. I asked him if he had any other plans at the time, he didn't, but I know i could suggest a few to him regarding his career.

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 10, 2016, 09:10:37 am
content: I have discovered a new phrase that applies to the meeting I've just witnessed.Atomic Facepalm.We have decided to stick with the USB Backup because "it has never caused a problem before" and because "this one time was probably just bad luck" and I was promptly labelled as having a "negative attitude" for saying suggesting that maybe the maintenance department should carry out periodic checks on the UPS (read "once a week, at least go and hit the self-test button on the front") and that me, the Tooling Engineer who happens to be good with computers probably shouldn't be doing maintenance work that he is not insured to do when we in fact have a maintenance department who's sole responsibility is maintenance.Well, obviously, since the tooling stores store their tools in the vending machine, checking the UPS for the vending machines is of course our duty...Well, good news, I'mnowinsured to do the testing............I don't get paid enough for this.--EDIT--OOH I forgot to mention; when I suggested we use the vending machine's network connection to do the backup in addition to the USB not only was I shot down in flames (I'm a "malcontent" apparently) but was told that it would "create a vulnerability in the network" somehow? They moved onto the next item on the agenda before I had a chance to inform them that a) it doesn't even have an external connection to the internet and b) is already connected to the network meaning any "vulnerability" that may exist is already there!

author: Cubdriver
author_rank: Unknown
timestamp: May 10, 2016, 03:35:16 pm
content: Sounds like a typical idiotic response from the higher ups. Just out of curiosity, how many times previously has the USB backup been used successfully?-Pat

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 10, 2016, 06:53:49 pm
content: QuoteGot it in oneAs far as I know, the last blackout we had was three years ago: but the UPS managed to shut things down gracefully.In other words: so far, the backups have never been successfully restored from USB following loss of the main data because there simply hasn't been a chance to - the first time we've needed it and it's failed. "Just a hiccup" of courseI really didn't like thead hominemattacks at the meeting though: usually I manage to stay on their good side and don't need to oppose their dictum; I've half a mind to take it to HR in fact, though to be honest I'm not sure I can be bothered - and I'm planning on moving on at somewhere sensible at some point.ThereAREsensible workplaces out there?Right?

author: Cubdriver
author_rank: Unknown
timestamp: May 10, 2016, 07:44:19 pm
content: So, as I suspected, the USB backup has been successfully used... ...never...It's been needed once, and it failed once.Based on their logic, since I've never had a flat tire it's perfectly acceptable if I drive around with a flat spare tire because "it has never caused a problem before".Given the 'logic' that often comes from higher ups, I sometimes wonder if promotions to management at some companies include mandatory lobotomies.-Pat

author: stmdude
author_rank: Frequent Contributor
timestamp: May 10, 2016, 08:03:06 pm
content: Somehow, that reminds me of this:

author: CatalinaWOW
author_rank: Super Contributor
timestamp: May 11, 2016, 01:40:57 am
content: You know, the OP had a shitty couple of days, but management might have been half right on this one. I don't know what the cost of weekly (or monthly or quarterly) tests of the backup system would be, but over the several years involved it might have added up to more than the two days of lost work and extra maintenance. I say half right because if you follow that logic they shouldn't have invested in the backup generator or the UPS supplies in the first place. And probably neither management or the worker bees have the information to make an optimum decision in cases like these. So the decision gets made on a gut feel basis by the guys with the power.Sometimes the best solution isn't the prettiest one. Like with USB thumb drives. They aren't reliable, in the sense of being the highest reliability storage mechanism out there. But three or five of them are dirt cheap, small and the odds of all of them failing are pretty darn low. Unless you are storing data on them for many years. But how often is mission critical data/software many years old? Often the refresh/update cycle is far shorter than the lifetime of those drives. And they beat the heck out of the highest reliability storage mechanism known to me. Characters impressed in mud and then baked into a ceramic form or chiseled into granite. Hand keying the information in from a slab of granite has no appeal at all.

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 11, 2016, 08:18:23 am
content: Even for a full power-off test the cost shouldn't be too much - on fridays we finish at 1430 on the shop floor, with office staff staying on until 1530: for the basic UPS self tests, it's just a matter of going around and hitting the buttons on the front. Even if you were powering off the UPS-protected systems it wont affect the office, meaning even if you done it once a month it'd only be a matter of ten minutes.The problem is that we've lost thousands of tool IDs and gigabytes of photographs which I luckily backed up two months ago to my own PC - but it's still a good two or three works to re-photograph everything that's been added since then and fixing the errors in the database. (as you can see, I'm putting my degree to good use!)If they had redundancy with the USB drives I'd be more than happy - two or three drives per machine wouldn't cost the world (hell, it wouldn't cost £20) - but they insist on just using that one drive.It just bugs me that the entire system is tied in with the network but doesn't actually make use of it! It also bugs me that when you point this out they throw ad hominems at you and hope you stop talking soon enough for them to go onto the next point on the agenda.Ooh - today's update is that they're not going to repair the generator but are going to sell it for parts and rely on the UPSs they don't want to maintain...I've given up arguing with them - better to not care about it and give up on worrying than to fight the system and stress oneself out! Wait, that's a terrible work ethic!

author: tszaboo
author_rank: Super Contributor
timestamp: May 11, 2016, 08:33:41 am
content: I had a car crash last week, when driving to work. Two broken doors, but I'm fine. And I'm solely responsible for it. Does this count?

author: Kjelt
author_rank: Super Contributor
timestamp: May 11, 2016, 10:12:17 am
content: The main cause is install and forget. Lead batteries for the UPS should preventively been replaced each 4 to 5 years, tested each quarter at least and so on.Everyone rants that this company did not do this but look at your own situation, when did you test your RCD's the last time (should be done each month according to the manufacturer) and this could cause something irreplacable, the life of yourself or dear ones. Or the smoke detector, yeah the battery is ok but does it work? Have you ever tested the thing with real smoke after you installed it 5 years ago? And so on. So a checklist should be made with regular to test and inspect items. We are all lazy and in a company it could be the "the other guy probably already took care of this" mentality that will end in disaster. Oh well we have a saying in dutch from the latin "Claudit eques stabulum, cum latro cepit equum" (The stable will be locked only after the horse thief has been caught),or something like that, but in our dutch case we loose a calf: "Only after the calf drowned the waterwell will be filled/closed"Now the big question is if the company acts to prevent this from happening a second time?

author: GreyWoolfe
author_rank: Unknown
timestamp: May 11, 2016, 12:31:42 pm
content: Quote from: daqq on May 08, 2016, 05:03:17 pmPriceless! I was also 'that guy' for my wife's family. I was told by SWMBO that I wasn't allowed to charge family members money. My plan was to act mildly exasperated every time they put a computer in front of me. Now they don't and life is good.

author: Kilrah
author_rank: Unknown
timestamp: May 11, 2016, 12:59:59 pm
content: Quote from: cprobertson1 on May 11, 2016, 08:18:23 amJust take half an hour to send a nice big email including all your analysis, with delivery confirmation and that you print a couple of backup copies of. That's the work ethic sorted, and the CYA as well should anything go wrong there's absolutely no more doubt where the faultdoesn'tlie.

author: Aeternam
author_rank: Unknown
timestamp: May 11, 2016, 01:52:06 pm
content: Quote from: CatalinaWOW on May 11, 2016, 01:40:57 amMaybe, this time. Because they were lucky.Imagine what would happen if a huge order gets messed up because data is corrupted or goes missing. Or payroll gets messed up. Or accounts receivable is wiped out. <Insert generic horror scenario here.>In today's computer driven business world not having a business continuity plan (this includeswell documenteddisaster recovery procedures, data backup plans andtesting and trainingall of this) is a recipe for disaster and can easily bankrupt a shop like this one. If disaster protection is too expensive in their eyes then they should rethink their business model.If management doesn't see this, they are not doing their job.

author: CJay
author_rank: Super Contributor
timestamp: May 11, 2016, 03:19:37 pm
content: Quote from: CatalinaWOW on May 11, 2016, 01:40:57 amNo.Not at all.What would have happened if the database didn't come back or was only repairable by hand entering thousands of transactions.A backup test costs nothing other than one knowledgable person's time.A total or partial but permanent loss from a database can cost a whole company's worth of staff's time. It's not comparable.(yes, I've done DR for a living, yes I've recovered data for idiots who believe backups are for the weak or don't need testing, one instance the backup reported success every day because all it was doing was a catalog of the tape.)

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 11, 2016, 08:40:13 pm
content: QuoteWe've lost about a fifth of the photographs (about 400) that will all need to be retaken - and we've lost maybe ten percent of the data entries; which all have to be re-inserted by hand once we figure out which drawer in the vending machine they come from and exactly what each item is (some don't have markings!)All in all, I reckon three or four weeks of hard work to get it all back up to scratch =/

author: CJay
author_rank: Super Contributor
timestamp: May 11, 2016, 08:44:47 pm
content: Quote from: cprobertson1 on May 11, 2016, 08:40:13 pmAnd if management don't realise that a proper backup regime is cheap after that then they need to be hauled up before their shareholders and flayed.

author: hammy
author_rank: Unknown
timestamp: May 11, 2016, 09:26:01 pm
content: Quote from: cprobertson1 on May 11, 2016, 08:18:23 amNo!If you cannot fix or improve the situation it is better for you to take the consequences.1.) Take care for your own health!2.) It's not your father's company!3.) Find another job some day!It's not on you to carry the burdens of the management decisions

author: bitslice
author_rank: Frequent Contributor
timestamp: May 11, 2016, 11:46:09 pm
content: This is inevitably something that will reoccur again and again within companies. IT people tend to only understand software systems and really only a sub section of how networks operate.What they probably also lack is an understanding of anything electrical like UPS's or how to make systems truly redundant from the plug to the end user.Yet there is nothing out there that would prepare them for this, it's nearly always done by a consultant or the last IT guy winging it on an ad-hoc basis with no budget and a limited knowledge of what is possible.Everything I know about IT is pretty much self taught, but by studying how other people do this on an enterprise level you can fill in the blanks in your knowledge and arrive at a solid system. Nobody out there teaches you any of this.As for backups, I've found the only limitation was on the available throughput of your systems.So if it takes an hour to backup onto a dumb USB stick, that still leaves a few hours to also backup on to tape, onto a hard drive somewhere else and maybe onto a cloud. You only stop running a backup when you need that network capacity for something else, unused Mbs are free and storage technologies are essentially unlimited too.A pity management can't recognise they are basing their decisions on an unknown level of ignorance.Mind you, when was the last time you ever saw an electronic design that was built with testability as a primary design goal, or any level of fail safe redundancy.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: May 12, 2016, 12:26:22 am
content: Quote from: Aeternam on May 11, 2016, 01:52:06 pmWhile I agree with the tenor of your reply, the specifics often become a formulaic recipe that DOES NOT provide value added. I have been involved in very large data backup programs and most ended up with costs out of line with the reliability provided. The problem is always an unforeseen single point of failure, or inadequate refresh rate on the data. Things like only one person involved in scheduling the personnel doing backup to remote site on a periodic basis. Then when the person scheduled to do the backup that time falls ill, and the scheduler also falls ill the backup doesn't happen. And Murphy is quite good at detecting this event. Not that I have ever observed this failure.Fortunately the interval was one week and the damages weren't that bad, just a little less than two weeks work by a team of 40 software engineers. Mitigated by the informal backups performed locally by many on the time.While I am unaware of any bulletproof systems the ones I have seen that work best seem to involve avoiding large singular data edifices and having distributed, redundant and independent backups. You end up with situation like the one described in this thread where there is a short term hit to operating capability, with full functional recovery over time. The system worked moderately well because in addition to the formal structure (generators, UPS, USB etc.) there was additional informal backup, some performed by the OP. This is often the case because the people closest to the problem develop a much better feel for what is important than those who are analyzing and developing the formal systems. This is why the informal mitigation mentioned above was so effective. Actual lost time was probably on the order of a few man-days.I will stick with my position that good data for creating the business plan almost never, if ever, actually exists. This would require good numbers for the probability of all possible data losses, good costs for each of these data loss cases, accurate costs and reliability for loss prevention strategies, and costs avoided for each loss prevention strategy (many strategies cannot totally eliminate losses, merely reduce them.) While the data protection plans I have seen often assign numbers to these items, the provenance of those numbers ranges from very solid to total phantasy. The same kind of problems that caused the Space Shuttle system to have an analytical reliability of less one failure in 10,000 while the observed failure rate was more like 1 in 50.

author: station240
author_rank: Unknown
timestamp: May 12, 2016, 12:48:11 am
content: Quote from: CJay on May 11, 2016, 03:19:37 pmSome companies have gone completely broke as a result of IT failures and no backup.Think things like the office burning down, and no offsite backup.Hence cannot pay staff, contact customers, pay taxes and suppliers.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: May 12, 2016, 04:22:51 am
content: Quote from: station240 on May 12, 2016, 12:48:11 amTrue. And some companies go broke when the IT failure includes failure of the backup strategy. Offsite wasn't offsite enough to deal with a regional catastrophe. Or wasn't reliable. Or something.Backup is always a good idea. But using the tragic cases as justification for a specific backup strategy just isn't good engineering. Just as survival of a specific failure is insufficient engineering justification for a backup strategy.We should remember that engineers are the ones who get to make decisions when there is not adequate data to completely inform the situation. It is one of the key distinctions between a scientist and an engineer. It should always leave us a little worried and a little humble.

author: cprobertson1
author_rank: Frequent Contributor
timestamp: May 12, 2016, 09:02:53 am
content: Quote from: CJay on May 11, 2016, 08:44:47 pmSee the problem is that management won't see me making all the fixes; to them it's just a work in progress - as far as they're concerned it's done and dusted!QuoteI'm officially trained as a scientist: the engineering side of my life started by accident... (that's a lie - I've been taking stuff apart since I was four (and putting it back together since I was six) - but I didn't officially have engineer in my title until a few years ago.@CatalinaWOW; station240; bitslice; Hammy; Aeternam; BitSlice (wow that was a long list!)In this particular case there are about 5500 images: totaling 2.5 gigabytes; and a table with 4000 odd entries.These are stored on the computer that runs the vending machine, and gets backed up once a week to the USB (literally an automatic copy+paste of the SQL Table and the image directories - so there's no redundancy in the backup).A few months ago I was tasked with making some changes to the back end of the database, and backed up everything so I could restore it myself in the off chance that I break everything - that's where I managed to get our backup from following the failure.In this particular case, the time and monetary costs of making a second copy onto a network directory (and as I mentioned, the vending machines are already connected to the network) would be negligible - and once it's on the network, it's automatically included in the daily backups as well (the daily backups are reduced to monthly backups after three months as well so we don't keep too much redundant data, but still quite a lot)The database side of things is literally a few megabytes and could easily be backed up to the network - but again, management are worried that it's a "vulnerability in in the network" (I think they read that in a magazine once and want to appear knowledgeable)The images on the other hand are the most awkward to back up because of their size - I can understand not wanting to put that on the network as the automatic daily backups will fill up a hard diskmoderatelyquickly; but at the same time, generating those images is very time consuming; each item needs to be manually taken out the vending machine, identified, photographed, numbered correctly, cropped, edited and then placed in the vending machines directory - so we could easily back that up to a couple of usb pens and just leave them nearby; even a manual backup once a month would be alright since we don't add in too many new items each month (maybe ten or twenty tops).I'm going to push for network backups of the database (which will be daily) and manual backups of the images maybe once a month or so - I'm in the process of being writing a brown-nosing email trying to emphasise the time and cost to recover from this and how little it will cost them to implement (and more importantly, that it isn't any more of a security risk than having the computer on the network like it already is!)@NANDBlog:Youch! Hope you're alright!Quote from: hammy on May 11, 2016, 09:26:01 pm1) sorted -- my health improved the minute I left the meeting room and it improved even more when I said "f*** it: if they can't be ***ed caring about the back up then I ****ing won't be caring about it either!" (that's not to say I wont try and rectify the situation; just that I won't be putting my heart and soul into it)2) No, but he does work here xD3) Already working on it! I'm a qualified biochemist, I'm sure I'll find something. (Yes, I'm a biochemist who became an engineer: it's a long story and it's REALLY boring so I won't bog you down with the details!)

author: CJay
author_rank: Super Contributor
timestamp: May 12, 2016, 11:39:59 am
content: Quote from: CatalinaWOW on May 12, 2016, 04:22:51 amMhmm, a well engineered backup solution shouldn't be informed by the tragic cases as you say but those tragic cases are the ones that convince non engineering staff to allocate the resource necessary which allows the techs to implement a well thought out strategy.Generally I find that a higher than expected percentage of the job of an IT tech is making it look like things are necessary, management will never implement a new email system or backup solution or UPS maintenance program etc. etc. until they can *see* a problem.