Newsgroup: it.lavoro.informatica
Titolo: Una storia di cluster

Ho appena mandato questa storia a The Daily WTF (
http://thedailywtf.com
), e 
ho comunque pensato di condividerla col newsgroup, nella speranza di 
alleviare il dolore.
Perdonate la lingua originale, ma di riscriverla tutta in italiano proprio 
non mi va... non dovreste avere troppi problemi con l'inglese voi 
informatici, comunque ;-)
Fatevi due risate se sopravvivete abbastanza a lungo alla nausea, e siate 
particolarmente prudenti, se mai dovesse capitarvi di mettere mano su un 
sistema di accesso condizionale per la TV digitale il cui nome inizia con 
una lettera greca.
Massimo
----------
I recently started working at a major Italian TV broadcaster, and since then 
I've been quite amazed at the quality (sic) of the software involved. After 
a month there, I'm absolutely sure they must be blessed by some unknown 
deity, because only miracles can make so many pieces of unstable and utterly 
incompatible software talk between each other and actually accomplish 
something (most of the time).
I witnessed programs that have full, clear, documented specifications, 
behave in totally different ways from what the specs say.
I saw four people manually checking and re-checking data consistency between 
seven different systems, because they can't trust the application which 
deploys the data to them, neither the systems that receive the data.
I stared in fear and trembling at servers which hosts thousands of euros' 
worth of applications and require you to log in and manually start the 
programs, because someone was just too utterly incompetent to know what a 
service or daemon is.
But nothing, ever, can reach the heights of the "cluster" implementation of 
***** (actual name length here), the software responsible for managing the 
conditional access system to pay-per-view TV programs.
This wonderful piece of s...oftware runs on Windows, using a SQL Server 
back-end; the people who developed it actually *know* Windows does have its 
own clustering system and how to use it (please take note of this), because 
they detail in their installation manual every single step necessary to 
build a working SQL Server cluster.
The application itself runs on an application server, or you can use two, 
for redundancy; when installed in this configuration, one of them becomes 
the active server, while the other one remains in stand-by mode, ready to 
start should the other server fail.
Ok, you should be thinking by now, so they created a standard Windows 
cluster application?
No, they didn't.
They created their own application, which runs on both servers and starts or 
stops the main services; each instance talks with the other one using a 
heartbeat link *which runs on the same network interface of the application 
server*.
Ok, well, maybe they just reinvented the wheel, and not quite in an optimal 
way, but it can work. Even Oracle implements its own clustering solution 
instead of relying on the operating system (did you notice a hint of sarcasm 
toward Oracle here?), so this can't be so bad, can it?
It can.
Remember what I was saying about some developers never having heard of what 
a Windows Service is? Here they are.
This wonderful "Redundancy Manager", along with the main application server, 
is a GUI application which *you* need to launch upon server startup. 
Manually. After logging on as Administrator, of course.
Ok, this is bad enough. But, at least, does it work?
It doesn't.
To be honest, the Redundancy Manager does quite a good job in starting the 
passive server when it can't contact the active one anymore. It does such 
good a job, actually, that it starts the passive server even if the active 
one just stops responding for 10-20 seconds.
The job at which it fails miserably is *stopping* one of the servers if, by 
chance or error, they become active together.
Let's restate it: if there is some transient network problem (cable 
unplugged, switch rebooted, power failure...), you end up with two active 
servers, which, of course, are definitely not designed to operate together, 
and will start kicking each other in their electronic butts in order to 
acquire some hardware resources they need. They *won't* recognize each 
other, and the Redundancy Manager, altough displaying a very warming 
pulsing-heart icon and a lovely "heartbeat established" description, 
wouldn't care less the two servers are both active and actively fighting for 
resources and (maybe) corrupting the DB. The situation *won't* stabilize 
after a while: they could go on forever, until one of them is manually 
stopped, maybe two days later if this happened on saturday.
And the final touch of this wonderful painting is... this isn't even a bug. 
The Redundancy Manager is *designed* to behave this way. It just doesn't do 
anything in such a situation, because it shouldn't happen. Never.
The manual states clearly that if one of the servers needs to be stopped 
and/or disconnected from the network, you should manually put it in passive 
mode before reconnecting and restarting it, or you can experience "service 
disruption".
Yes, you most definitely can experience it.

 informatici, comunque ;-)
Agghiacciante. E io che di mestiere metto su cluster Windows
rischio di rimanere disoccupato perché in giro ci sono tali
peracottai? Che tristezza!
Fra