This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

se (OKTHE) | ay (OXI) os secs : +
After receiving mail, visited a small division of the Air Force in

Alexandria, Virginia." Testimony Tr. at 58." _ testified that the site had
been pre-selected by the vendor. "° _| Testimony Tr. at 43 (“[T]he sites were
provided by the vendor. I don’t recall if it was Apple or Cloverleaf that provided the
sites.”). "described the site visit as follows:

I learned that, you know, when you go to a site like that,
number one, we really can’t get down to a system and look
at the data because of the security and all that stuff. ... But
we did go into computer room. We saw the equipment was
up. We talked to the engineers that were there and they
raved about the equipment. They thought it was great. ...
But they were very small. I mean it wasn’t for Air Force
wide, It was a very small organization right down in
Alexandria.

August 17, 2010!°° | Testimony Tr. at 26.

As discussed below in Section III, approximately five months after Cloverleaf
was purchased, an OIT Specialist,° was tasked with its
implementation. Before he even took the equipment out of its crates” |did a
“theoretical” analysis of its performance and concluded, “I cannot see how Cloverleaf
will work.” See January 10, 2009 e-mail from! ta _| Exhibit 28.[°° |testified
that the analysis" performed should have been done before Cloverleaf was
purchased:

Q: ...[TJhe theoretical exercise that you and” | are
discussing here that you’re going through, is there any
reason that this couldn’t have been done prior to the
acquisition?

A: Goshno. That could have been -- that should have
been done way -- the theoretical -- in all honesty, the
theoretical should have been done before you even do
a selection of who you want to bring in to test. ...
[Y]ou should have these types of discussions early so
you can say, okay. This thing, theoretically, based on
the numbers they’re giving us, doesn’t meet our
requirement. Don’t waste our time.

(Ney

17


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

Justification and Approval for the first time during her testimony, she described it as “the
most pathetic [justification] I’ve seen.” Jd. at 35. She also testified that she would not
have approved the Justification and Approval. Jd. at 3921

In contrast to" view," |maintained that OIT had adequately justified
a sole-source contract: “I would have approved it again, because | thought they made a
good case. ... And [OIT] made a convincing argument to me in this [Justification and
Approval].”°""° Testimony Tr. at 53. See also, id. at 49 (“[T]hey’re making a
statement to me saying that Apple Computer is currently the only seller of Cloverleaf,
and the required Promise RAID storage to the U.S. Government, that’s a very strong,
statement to me. ... So, that would lead me to say, ‘Okay, if that’s where you guys want
to go, and you’ve done the research, and you’re telling me that,’ I would probably say,
‘Okay.””); Id. at 61 (“{T]hey’re telling me, ‘I’ve got a need,’ and it’s a new thing, ...And
so, I have nothing in here to tell me that I should be competing this.”).

When he reviewed a draft of the Justification and Approval during his testimony,
testified that he also believed the draft Justification and Approval was sufficient
to justify a sole-source contract:

(exmxey

Q: [When you saw this draft, did you feel this -- I’m
trying to get a sense of your reaction and your
comments. Did you feel that this generally was
sufficient for sole source justification and just you had
a few comments on how to improve it or did you feel
like it was inadequate as written to justify sole source
contracting?

A: I believe that, from what was here, would support a
sole source, but I believe it needed to have some more
descriptions built into this.

exexey

i
Testimony Tr. at 61-62.77

a At least two significant improprieties identified by the OIG in this matter were previously

identified by the OIG in another matter. See OIG Management Alert Report No. 469 dated August 10,
2009, attached hereto as Exhibit 45. In April and May 2009, the OIG conducted a limited-scope review of
the Commission’s sole-source contract for Microsoft Premier Support Services as a result of an anonymous
complaint received from the Government Accountability Office. /d. That review related to a sole-source
contract awarded to Microsoft on April 27, 2006, by OA and OIT. /d. With respect to the Microsoft
award, the OIG found, inter alia, that the Justification and Approval did not support the award of a sole-
source contract and that the Justification and Approval was not reviewed or approved by the competition
advocate despite the fact that the total contract value was in excess of $1,000,000.

2 The draft Justification and Approval that” ‘reviewed during his testimony was an
abbreviated version of the final Justification and Approval. See Exhibit 46; cf. Exhibit 42. When
had reviewed the same draft Justification and Approval in July 2008, he had commented “Why
Cloverleaf?” Exhibit 46 at 1.°"%° explained in his testimony that he had made that comment because
the draft Justification and Approval “needed to give further explanations to why Cloverleaf was needed,

38

(xe)



This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

else could sell me this Cloverleaf solution. I found nobody
else besides Apple ....

rexTxcy

August 17, 2010 Testimony Tr. at 54.

In essence, the Justification and Approval stated in conclusory fashion, without
any support or explanation, that OIT wanted to buy Cloverleaf, and then stated that the
SEC could only buy Cloverleaf from Apple. Jd, That rationale for the Justification and
Approval was first provided by °° _ jin a July 14, 2008 e-mail re “So I believe
we only need the cloverleaf [sic] justification, which is easy since it can only be
purchased via Apple professional services.” Exhibit 44.

After explaining generally the circumstances that justify a sole-source acquisition
during her testimony," _|was asked to review the Justification and Approval that
was approved for the Cloverleaf acquisition. °° Testimony Tr. at 35. Her
immediate reaction was as follows:

[Question] Number 5: 5 says, “Demonstration that the
proposed contractors’ unique qualifications or the nature of
the acquisition requires use of the authority as cited.” [’ll
be honest and blunt: 5 is the most pathetic description I’ve
seen. It is completely inadequate under this J and A in my
opinion. ... It says nothing. ... What it’s saying is that
Apple is the only seller of Cloverleaf. So what? It doesn’t
say why Cloverleaf is the only product that can meet your
requirements described under 3. All right. So number 5
should have a much more in-depth explanation as to why
the requirement can only be met by this product -- not that
Apple’s the only one that sells Cloverleaf.

pene

Testimony Tr. at 35-37.
© also noted that the Justification and Approval did not include an
estimated value of the acquisition, which is also a violation of the Federal Acquisition
Regulation. See FAR 6.303-2(a)(3); °° Testimony Tr. at 39 and 41; Exhibit 42 at
1, © |estified about the Justification and Approval’s lack of an estimated value as
follows:

Q: ...[OJne thing that the document does not have is an
estimated ...price. It’s my understanding ... that that
is actually, under item three, something that is
supposed to be included.

A: Itis. It says, “Estimated value,” and I did not pick up
the fact that it wasn’t there, but yes, it is supposed to
be there. And just about every time it is.

34


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

Id.

(OxTKC)

On February 26, 2009,

e-mailed an Apple representative the following:

It looks like the “plan” was to get the SEC to buy the
Xserves for LA and Denver at a later time. I need to know
if any other items, that would have “normally been
included”, were left off of the three purchase orders that
were sent to Apple for this project.

Exhibit 55.
On March 13, 2009," e-mailed!” —_ the following regarding Cloverleaf:
Issues / Concerns thus far:

* Two Apple Xserves, for LA and Denver, were left off the
original order by Apple to keep costs below $200K. These
servers will need to be replaced before we install in LA and
Denver.

* None of the Promise storage was ordered with GBICs.
They will need to be purchased before installation.

* It is looking like additional Promise Storage will be
needed for LA and Denver to give enough spindles (and
thus enough IOPS) to the Cloverleaf to allow for the
snapshots to work properly. Exactly how much storage is
TBD? If so, this could be a major problem for LA
where space and power are already an issue.

It is clear to me that Cloverleaf had very little information
about the server and storage problems faced by the SEC, or
maybe they and Apple knew the issues and choose to
ignore them to get their foot in the door. Either way, I do
not see any evidence that a clear design for LA - Denver or
OPC - ADC, based on solid requirements was produced.
Essentially they drove for a dollar amount. ...

Exhibit 56 (emphasis and ellipsis in original).
O°" |testified that the SEC’s $200,000 Cloverleaf equipment did not include all
the necessary items to install Cloverleaf and that he had to subsequently purchase

additional equipment. August 19, 2010” Testimony Tr. at 65." (confirmed that
additional equipment was needed to install Cloverleaf:

43


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

[Cloverleaf] was purchased more as a test to see how well

it performed compared to other storage that we usually
purchase. ... We have a lot of storage that’s obviously used
for various reasons around the Commission. I believe that
most of what we have is EMC storage. EMC is probably, if
it’s not the most expensive storage you can buy, it’s
certainly well up in cost. So the intent was to try to save
some money for us, for the taxpayers, whatever.

I was looking for cost savings. ... The reality is whether it
was this technology or another technology. Mostly, I think
my drive was to say why do we keep buying the same
stuff? Why aren’t we looking at [storage solutions other
than EMC]? ... Why aren’t we looking at, maybe,
Netapps? They’re probably just as expensive. ... [With
Cloverleaf] the cost difference [was] just too hard to resist.

Walker Testimony Tr. at 13-14, 53-54.5

After the January 28, 2008 meeting with the Cloverleaf representative,
also excited about purchasing equipment from Cloverleaf. See Exhibit 17. In fact, a
January 29, 2008 e-mail exchange between!” land Walker suggests that they may have
already decided to purchase Cloverleaf. /d. One day after the meeting with Cloverleaf,
OX ‘e-mailed Walker:

If the timing is right, we can have this [Cisco switch] as
part of the design at Station Place. We will have
Consolidated network/storage/backup switch, we will have
Virtual server (Vmware and Scalent), we will have virtual
storage (Cloverleaf) .... How exiting [sic] ...

dd, (ellipsis in original). Walker responded, “Sometimes things seem to come together at
the same time. Maybe this is one of those times.” Jd.

s Walker also testified that his desire to try out Cloverleaf was influenced by a comparison test that

(Was.gonducted at a certain technology fair. Walker Testimony Tr. at 48-49. As discussed below, fexxe
OIT Specialist, was tasked with installing and operating the equipment that was part of the
Cloverleaf acquisition. °“ testified that he spoke to the person in charge of the comparison testing that
Walker referenced. August 19, 2010(°” Testimony Tr. at 56, 60. [also obtained all of the data for
the results of that comparison. 12° testified that he leamed that Cloverleaf was not the least expensive
product that was tested. /d. at $7. More importantly” learned that the test data that was used in the
comparison did not require a solution with high IOPs, unlike the SEC’s data. /d. at 57-58.

11


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

have been approved. In addition, the purchase of Cloverleaf equipment before the
written justification had been approved by the contract officer violated Section 253(f)(1)
of the CCA and Subpart 6.303-1(a) of the FAR, and the failure of the written justification
to include the estimated cost of the acquisition violated Subpart 6.303-2(a)(3) of the
FAR.

The OIG also found that OIT violated Subpart 15.201 of the FAR when it shared
budget information with Apple in order for Apple to tailor the first of the Cloverleaf
orders within the SEC’s budget parameters, even though that order omitted essential
equipment that the SEC was subsequently forced to purchase, and tied that order to two
larger orders that were placed one week later. OIT.also violated the SEC’s CPIC.
procedures by waiving a pre-acquisition review of the Cloverleaf project.

The OIG found that as a direct result of those numerous violations-of the CCA,
the FAR, and the SEC’s CPIC procedures, the SEC invested approximately $1 million in
technology that immediately failed to perform its intended function. This report is being

provided to the Deputy Chief of Staff, Office of the Chairman, Commissioner Elisse
Walter, Commissioner Luis Aguilar, Commissioner Troy Paredes, General Counsel, the
Chief Operating Officer, the Chief Information Officer, the Executive Director, the
Associate Executive Director, Office of Administrative Services, and the Associate
Executive Director, Office of Human Resources so that appropriate action (which may

include performance-based action and/or training) is taken with respect to Walker, °°
(oy7XC) and (XTC)

The OIG also recommends that OIT institute appropriate procedures to insure that
any significant technology acquisition undergo an adequate pre-acquisition review of its
technical merits and compatibility with the existing information technology architecture.
Finally, the OIG recommends that OAS institute appropriate procedures, including
training of its contract specialists, to ensure that future procurements are done in accord
with federal statute and regulation.

(exrxe)

Subm Date: la Ll Y [ Qed
Conet Date: © 12/14 / o-
Approved: WA GLE Date: (uff Ife

CH. Bavid Kote

53



This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

at headquarters, relative to its expense, how much
would it cost to go out and get a piece of hardware to
do just that?

A: ... Right now it’s supplying 200 terabytes of data. ...
I’m just going to use a calculator. ... So 200
terabytes and it roughly costs a million dollars ...
5,000 a terabyte. That’s what it cost. It would have
cost half that. ... [I]t was $5,000 a terabyte and it
would have been 2,500 or 2,600 a terabyte. ... So it
costs more plus there is a lot less performance.

(eN7XC)

Id. at 62-63. also explained that the Promise storage arrays have caused problems,
including a loss of backup data: “[As part of the Cloverleaf acquisition, OIT] bought
really cheap performance arrays. ... We actually had instances where we lost, because
we had failures on the Promise side, we actually lost backup data.” /d. at 47-48.

er lexpressed a completely different assessment of the Cloverleaf acquisition.
He testified:

Q: ... What is your understanding of whether or not the
Cloverleaf technology was successful in meeting the
Agency’s need that it was purchased for?

A: My understanding of the Cloverleaf technology is
brought into Commission with pricing -- no. With the
— it met the requirements that we are using today at the
ADC for backing up the data and storage. It solved the
problem we had at the ADC for the backing -- the
backup the data. It also brought in a competitive
advantage to the SEC by looking at different solution
that is less expensive than the current storage and
backup solution we have using EMC in the
Commission. And it helped us in the engineering
process to go out and look for more alternative. Don’t
stick with what we have. You think if the product we
have is expensive and the market out there is very
competitive, then go out and find alternative and you
will find the good products that meet your requirement
with less price. Overall, it helped the Commission in
term of technology investment.

(NTXC) .
Testimony Tr. at 48-49.

Although |!” claimed that Cloverleaf had been a success, he acknowledged that
Cloverleaf is not providing all of the backup function for the SEC; that “[t]there is no

51


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

equipment. See Exhibit 62. Regarding the first option, using Cloverleaf for data backup
in the LA and Denver offices as intended when it was purchased, wrote, “This
option would require a substantial investment in additional storage to act as a disk target”
and “[t]here are technical risks that [would have to be] addressed.” /d. at 1. The second
option, which _' preferred, was to “repurpose” Cloverleaf for some unspecified
purpose “at the OPC and the ADC.” Jd. °° recommendation for “moving
forward” with a solution for backing up data in the regional offices was to “turn toward
the use of NAS (Network Attached Storage) devices,” Jd. at 2. °° _ estimated that
the total cost of a NAS solution for the LA office was “$66,000 - $99,000” and pointed
out that “[t]he Cloverleaf iSN that was purchased for LA last year cost [approximately]
$100,000, and that was without any storage for that site.” d.°°

x” The March 2010 OIG Audit found, inter alia:

[I]n March 2009, the sponsor discovered that the [Regional Office Backup
Project] had major problems with overheating and performance. ... [T]he
equipment purchased to improve the regional office backup capability did
not have the adequate performance levels needed by the regional office
and, in fact, would lower the regional offices’ case system performance
instead of improving it. [In addition,] [t]he IT security group rated the
project as a high risk due to the problems identified during the security
testing. Where significant problems are found, the sponsor of a project is
required to go back to the PRB immediately .... In this case, the testing
highlighted problems with both (a) performance and delivery expectations;
and (b) documented technical and operational risks and expectations, either
of which would constitute a significant baseline change.

However, ... contrary to the prescribed process, the sponsor did not submit
a change request in March 2009. Instead, the sponsor decided in April
2009 to reuse the equipment to support the [ADC Becomes Primary]
project and cancelled the Regional Office Backup project. Only in August
2009 did the sponsor finally return to the PRB, one year and two months
after the initial approval of the project and five months after the problems
had been discovered during the performance testing, to submit a change
request and to inform the board that the pilot did not work, the project was
being cancelled and the equipment purchased for the project was being
used to support another ongoing IT project.

In this case, the PRB was not afforded an opportunity to state whether it
approved the equipment being used to support the [ADC Becomes
Primary] project because the sponsor decided to cancel the project and re-
purpose the equipment without notifying the PRB. ... Consequently, the
SEC has expended $200,000 for equipment that did not work for its
intended purpose, and the regional offices still have a storage problem that
needs to be addressed.

March 2010 OIG Audit Report at 9-10,

49


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

(XX) |testified that he found it “odd” that OIT decided to purchase Cloverleaf
and send it to the LA and Denver regional offices as a “proof of concept” test:

I thought [it] was a little odd was that we were purchasing

the equipment for a proof of concept. ... We were

purchasing the equipment and we were doing a proof of

concept in L.A. and Denver. I thought that was just a little

bit odd.
August 17, 2010°°” Testimony Tr. at 23. "described the Cloverleaf
acquisition as “more of a production system than a proof of concept system.” August 17,
2010!" | Testimony Tr. at 24. | explained further:

A: Well, this was a little different. I’ve done proof of
concept before and usually the [vendor] provides the
equipment. ... [T]hey provide the equipment and
everything so you can bring it in and try it out. And
you’re not promising that you’re going to purchase the
equipment or make any kind of promises. You just
want to see where it fit, see where it would work and
how it would work in your environment. .... In this
particular instance, we didn’t do that. We actually
bought the equipment for proof of concept.

Q: Why? Why the difference?

I don’t know. I mean I don’t know why we decided to
buy it.

Q: So in your experience is this the only time that the
equipment’s been purchased for a proof of concept
project, and, you know, funds have been committed, et
cetera, before it’s ever tested?

A: Yes. .... 1 personally think if we’re going to call it a
proof of concept that Apple should have provided
equipment without us paying for the equipment, and
we tested out in our environment. [In addition,] I
disagreed with [Cloverleaf] going out into the regions
(as a proof of concept]. ... We should have done a
proof of concept here at headquarters.

(exrxey

testified that OIT could have easily tested Cloverleaf on a no-cost basis:

Q: Is there some reason you think that wasn’t possible?
Would that have technologically not been possible, or

21


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Generals approval.

The OIG found that” assessment of the Justification and Approval was

accurate. Specifically, the OIG found that the Justification and Approval did not, even on
its face, justify the sole-source acquisition of Cloverleaf.

C. OAS Purchased Cloverleaf Before the Justification and Approval was
Approved in Violation of the Competition and Contracting Act and the
Federal Acquisition Regulation

The Competition and Contracting Act and the Federal Acquisition Regulation
require that a contracting officer may not place a non-competitive purchase order until
the written justification has been approved. See 41 U.S.C. § 253(f)(1); 48 CFR pt.
6.303-1(a).

The three purchase orders for the Cloverleaf acquisition were signed by" [on
August 7, 2008. See Exhibits 47 and 48. However, the Justification and Approval was
not approved by” |until August 11, 2008.2 See Exhibit 42. In fact, as of August 4,
2008, only three days before” |signed the purchase orders," | had not received a
revised draft Justification and Approval that addressed such fundamental comments as
“Why Cloverleaf?” See Exhibits 46 and 51. On that day, [°° _| e-mailed "°
explanation of his comments and acknowledged that he did not have “a full
understanding of {Cloverleaf or Promise].”** See Exhibit 51.

(°° _\testified that it was improper to have signed the purchase orders four

days before the Justification and Approval was approved:

Q: [The Justification and Approval] is dated August 11,
2008. And the three purchase orders covered by the
justification ...all dated August 7, 2008. Do I

why specifically Cloverleaf versus some other hardware or software was needed. ... We didn’t say why we
needed Cloverleaf. We just said it was compatible.” °° [Testimony Tr. at 61.

2% The Justification and Approval had not been annroved, in writing or otherwise, before the
purchase orders were signed. On August 8, 2008, (PX eamailed (7%) suggested changes to the draft
Justification and Approval. See Exhibit 49. On August 11, 2008, °° [e-mailed the final Justification

and Approval to fener [flor {her] review and signature.” See Exhibit 50.

24 A July 24, 2008 e-mail from!" to" ‘evidences! _|fundamental lack of
understanding regarding why OIT wanted to purchase Cloverleaf. See Exhibit 46. In that e-mail,
attached his comments to a draft Justification and Approval and explained:

(exTKC)

Hint, need to talk a little bit more about why Cloverleaf is needed and why
we can only get that from Apple. You have support for Apple because we
are building a parallel infrastructure (my words) to enhance infrastructure
security.

Id.
39


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

The OIG investigation found that the Cloverleaf acquisition violated several
provisions of the Competition and Contracting Act (“CCA”) and the Federal Acquisition
Regulation (“FAR”). The SEC’s Office of Administrative Services (“OAS”) approved
the Cloverleaf acquisition on a non-competitive basis. However, the OIG found that the
justification cited for the award on that basis, “only one responsible source ... will satisfy
{the SEC’s] requirements,” was inapplicable. Consequently, the Cloverleaf acquisition
violated the central tenet of the CCA and the FAR; that “except in the case of
procurement procedures otherwise expressly authorized by statute, an executive agency
in conducting a procurement for property or services shall obtain full and open
competition through the use of competitive procedures.” 41 U.S.C. § 253; see also, 48
C.F.R. pt. 6.3 (“requir[ing], with certain limited exceptions, that contracting officers shall
promote and provide for full and open competition in soliciting offers and awarding
Government contracts”).

Moreover, the OIG found that the Cloverleaf acquisition was not reviewed and
approved as part of a process to promote competition as required by statute, another
violation of the CCA and the FAR. The OIG further found that if the Cloverleaf
acquisition had been reviewed by the SEC’s Competition Advocate, it would not have
been approved.

In addition, the written justification for the award of the contract on a non-
competitive basis did not include the estimated cost of the acquisition, as required by the
FAR; the Cloverleaf equipment was purchased before the written justification had been
approved by the contract officer, another violation of the FAR; one of the requisition
requests and purchase orders for the acquisition was improperly split into two orders; and
the OIG found that OIT violated the FAR when it shared budget information with Apple
in order for Apple to tailor the first of the Cloverleaf orders within the SEC’s budget
parameters, even though that order omitted essential equipment that the SEC was
subsequently forced to purchase, and tied that order to two larger orders that were placed
one week later.

The OIG also found that OIT violated the SEC’s CPIC procedures by waiving a
pre-acquisition review of the Cloverleaf project. As a result of that violation, the
approximately $1 million acquisition of a technology that had never been used at the SEC
from a company with no proven track record in the information technology industry was
never reviewed on its technical merits.

Finally, the OIG found that as a direct result of those numerous violations of the
CCA, the FAR, and the SEC’s CPIC procedures, the SEC invested approximately $1
million in technology that immediately failed to perform its intended function. Asa
result of its investigation, the OIG is recommending that OIT institute appropriate
procedures to ensure that any significant technology acquisition undergoes a pre-
acquisition review of its technical merits and compatibility with existing SEC systems.


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

(exTKe) THC)

Testimony Tr. at 75. °° |also agreed that OIT’s belief that Apple could provide
a less expensive storage and backup solution should have been tested in a competitive
bidding process:

But this doesn’t tell me that [OIT is] looking to get a
cheaper price. It isn’t that I would have rejected it if 1 had
known that, but I would have made them do a competition
to prove themselves right. And I’m doing that a lot more,
because they will say, “Oh, this is the only one.” And I
will look for the little holes in it, and I will say, “Well,
okay, that’s what you say, but give me your specifications.
I am going to compete it, just to prove you right.” And
that’s when I find out whether they really are, you know --
whether their specs are really true or not.

eo Testimony Tr. at 52.

According to (eX7XC)
Cloverleaf acquisition was “pathetic.” See
Justification and Approval stated in relevant part:

OIT’s attempt to justify a sole-source contract for the
ene) Testimony Tr. at 35. The

2. Nature and Description of Action Being Approved:

The Commission has a sole source requirement to
implement an intelligent storage network solution that will
manage all current and future network attached storage, and
that is vendor and technology neutral (heterogeneous), and
provides data protection with periodic snapshots and offsite
data replications.

This intelligent storage network solution requires OIT
to identify, test and deploy a new virtualization technology
that is not currently used at the SEC. This solution
provides true heterogeneous storage management, and disk-
based backup and recovery

3. Description of the supplies or services required to meet
the needs of the SEC and the estimated value:

OIT will acquire services, hardware and software from
Apple Computer Inc. The Promise RAID Storage system
and the Cloverleaf Intelligent Storage Networking system
will be acquired through Apple. The Promise RAID
storage system provides a highly reliable and scalable
solution that meets the SEC enterprise storage

32


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

% ‘Testimony Tr. at 71-72. feiterated that the “theoretical review that [was done]
post-acquisition [absolutely] could have and should have been done before the
acquisition.” Id, at 77.

Irrespective of the fact that OIT did not perform a pre-acquisition technical review
of Cloverleaf, it could have tested the technology as a proof of concept. A “proof of
concept in engineering methodology is try to have a small scale of a certain technology or
certain solution and to determine if that solution will work in a bigger environment.”

(2° |Testimony Tr. at 57. Generally speaking, an information technology proof of
concept project has two characteristics: (1) the vendor provides the equipment at no cost
for a trial evaluation and (2) the trial test is done in a lab using test data, not in a
production environment. See August 17, 2010{%° {Testimony Tr. at 20-21; August
19, 2010" Testimony Tr. at 8-9. Initially, “trying out” Cloverleaf as a possible
solution to the SEC’s backup problems was supposed to have occurred on that basis. See
April 29, 2008 e-mail from |to!””_|attached as Exhibit 30,”

Testimony Tr. at 17; August 17, 2010'%° ‘Testimony Tr. at 18-19.”

On February 20, 2008, the Cloverleaf representative from the January 28, 2008
meeting e-mailed!"

During our meeting I offered to host you in our Woodbury
office or bring an evaluation system into your lab. Can we
have a short phone call this week to talk about next steps?

Exhibit 20. However, as discussed below, OIT did not pursue the opportunity to try out
Cloverleaf at no cost to the SEC.

On April 9, 2008, the Cloverleaf representative e-mailed er copies of
Cloverleaf’s mutual non-disclosure agreement and its product evaluation agreement and
stated:

It was nice to see you again this afternoon. Pursuant to our
discussion, [an SEC contractor] asked me to send these
documents to you for review and signature. I need to get
these in place in order to provide you with an evaluation
iSN. The team suggested that early May was probably the
right time to get this started.

Exhibit 31.

(enTHe) (OxTKe)

On April 29, 2008, e-mailed OAS Contract Specialist:

Here is another product with which OIT would like to
perform a no cost evaluation. This is a data backup/storage
management product from a company called Cloverleaf.
The evaluation would be performed in the OIT test lab.

18


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

(exrxe)

Testimony Tr. at 33; Testimony Tr. at 21; August 19, 2010" Testimony Tr. at

87.

As®*° explained:

If I’m going to, with Cloverleaf, put a system dead in the
middle of my storage architecture, you know, that staying
within the Windows box, the Unix boxes and the storage,
I’m going to want to consult the stakeholders. Okay? The
stakeholders at a minimum are going to be the people that
tun the windows boxes and the people that run the
storage... I’ve heard that that was one problem in the
process here; all the way up to the actual acquisition, that
within the OIT work chart the storage group... were not at
the table at all, were not consulted.

(BNTNCY

August 19, 2010

(BxTKCY

Testimony Tr. at 33.
reaffirmed:

{I]n the case of Cloverleaf, the backup group did the whole
design and excluded the storage group. And that’s one of
the key elements, I think, that was a mistake. ... [T]he
whole Cloverleaf piece came because we have backup
problems. So they did this whole design, or what have you,
based on solving a backup problem, and it completely
changes the storage environment, but the storage folks
weren’t included.

[T]he best engineering efforts go through a peer review
process and have all the stakeholders at the table. If you
don’t have the stakeholders involved, then it becomes very
lopsided one way or the other.

(eX7XC)

Testimony Tr. at 13-15.

D. Before the Part of the Cloverleaf Acquisition that was Intended for Data
Backup in the LA and Denver Offices was Made, OIT Decided to
Purchase another $773,000 of Cloverleaf and Associated Apple
Equipment for a Different Purpose at SEC Headquarters

As discussed above, OIT’s desire to try out the Cloverleaf technology quickly
evolved from a no-cost proof of concept test to a $200,000 plan to install the equipment
in the LA and Denver regional offices. In addition, very shortly after the $200,000
purchase, OIT decided to purchase an additional $773,000 of Cloverleaf equipment for

24


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

Q:_... [D]id this ever happen, what you were telling!"
(9 you would like to do, which is perform a no cost

evaluation of Cloverleaf? ...

A: I don’t believe that it was, no.

Q: ... Whether there was cost associated or not, do you
recall whether or not a small-scale test and evaluation
in the OIT test lab was ever performed before the
ultimate acquisition?

A: I don’t recall that happening, no.

... Do you remember why [not]?

A: No, I don’t. ... I don’t know why.

Q: ... I think this would strike anyone as a sensible way
to proceed with testing that new technology before
embarking on a large acquisition and/or deployment in
the production setting. Would you agree with that?

A: I would agree.

Q: Do you know why that wasn’t the way, ultimately, the
Cloverleaf acquisition occurred?

A: I don’t recall. ...

Q: ...[M]y understanding is that there was a proposal and
a plan to purchase the equipment that is referred to
here on [requisition request] 2455 for $200,000 from
Apple, and that was going to be implemented in the
LA office and the Denver office.

A: That’s correct.

Q: And was going to be their backup.

A: It was intended to be a proof of concept for this backup
capability.

©") | Testimony Tr. at 18-19, 47.

20


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

requirements. Cloverleaf provides storage management,
high availability, and disaster recovery for SEC business
continuity. The combined Cloverleaf and Promise solution
gives the SEC a cost-effective storage network with a
footprint scalable for the Regional offices, Operations
Center, and Alternative Data Center.

OIT will acquire the installation and consulting
services from Apple Computer Inc. Hardware and software
required to manage the Promise and Cloverleaf
environments will also be purchased. OIT anticipates
follow-on acquisitions for similar hardware and software
compatible with Promise and Cloverleaf environments
which this justification and approval will apply.

4. Statutory Authority for Other Than Full and Open
Competition:

The statutory authority permitting other than full and
open competition is 41 U.S.C. 253(c)(1) FAR 6.302-1(a)(2)
— Only one responsible source.

5. Demonstration that the proposed contractor’s unique
qualifications or nature of the acquisition requires use
of the authority cited:

OIT has performed online market research and has
received vendor notification to determine that Apple
Computer Inc is currently the only seller of Cloverleaf and
the required Promise RAID storage to the US government.

6. Efforts to ensure that offers are solicited from as many
potential sources as possible:

Apple is the only manufacturer / vendor capable of
fulfilling the requirements given above.

See Exhibit 42 at 1-2.
(0 described his effort to articulate a justification for the Cloverleaf
acquisition as follows:

I believe, what I was saying here, and I may have answered
it incorrectly, but I believe what I’m saying here, we had
already decided at this point that we wanted the Cloverleaf
solution. ... And I went out and I tried to see if anybody

33


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

storage needs for the regions; or, I’m sorry, the back-
up needs for the regions?

A: .... Absolutely. But it was different and EMC at the
time was viewed as very expensive.
August 17, 2010°"° ‘Testimony Tr. at 16.
poe testified:

Q: ... [What companies and products do you recall,
other than Cloverleaf, having looked at for that
[backup] function ...?

A: One, again, was Data Domain, a company that we
looked at. We brought their product into the lab.
Another product that was offered by EMC, and I don’t
recall the name of that product. It was similar, I
believe, to the functions that we thought Cloverleaf
might provide.
(&X7XC)

Testimony Tr. at 15.

According to OIT e-mails, Cloverleaf, VMware, Scalent, Data-Domain, and
Exagrid offered some solution to the SEC’s backup problems. See Exhibit 25 at 3;
Exhibit 26 at 1. A June 19, 2008 e-mail discussed the rationale that resulted in Cloverleaf
being chosen for the “[bJackup redesign:”

Go)

15


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

6.303 Justifications.
6.303-1 Requirements.

(a) A contracting officer shall not commence negotiations for a sole source
contract, commence negotiations for a contract resulting from an
unsolicited proposal, or award any other contract without providing for
full and open competition unless the contracting officer—

(1) Justifies, if required in 6.302, the use of such actions in writing;
(2) Certifies the accuracy and completeness of the justification; and
(3) Obtains the approval required by 6.304.

6.303-2 Content.

(a)Each justification shall contain sufficient facts and rationale to justify the
use of the specific authority cited. As a minimum, each justification shall
include the following information:

(2) Nature and/or description of the action being approved.

(3) A description of the supplies or services required to meet the
agency’s needs (including the estimated value).

(5) A demonstration that the proposed contractor’s unique qualifications

or the nature of the acquisition requires use of the authority cited.

(6) A description of efforts made to ensure that offers are solicited from
as many potential sources as is practicable, ....

(7) A determination by the contracting officer that the anticipated cost
to the Government will be fair and reasonable.

(8) A description of the market research conducted ... and the results or
a statement of the reason market research was not conducted.
6.304 Approval of the justification.

(a)Except for paragraph (b) of this section, the justification for other than full
and open competition shall be approved in writing—


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

foxx) |OAS Management and Program Analyst, testified that a
Justification and Approval was inadequate if it only articulated a “preference” for a
specific product or vendor:

Q: [For example,] if the need was ... a thousand PCs and
... [had a preference for Dell. ... 1 thought they would
be the most cost effective ... But the need was for a
PC. ... I gather from your explanation that you’d have
to justify the need for the Dell as opposed to just the
need for the PC. I couldn’t write just the sole source
justification that says I need 1,000 Dell computers, and
Dell is the only one that makes Dell computers.

A: Right. You would have to say what is it about Dell
Computers that is required -- not a preference -- as you
made that distinction. ... Why is Dell the only one
that’s going to work for you?

(exTKCY

Testimony Tr. at 23.

With specific reference to the purpose of the Cloverleaf acquisition, °°"
described as “premature” a decision to buy Cloverleaf, if OIT was aware that other
vendors had products that could solve the SEC’s backup and storage problems, but “the
Apple vendor made a pitch that they could do the same, that they had a product that
would also work, and .... [OIT] liked the sound of the Apple product.”
Testimony Tr. at 26-27.°°" explained further:

So you always have to go back to just the need. ... [T]he
need is back-up storage and right now they’ve identified
two vendors that have that capability. Yeah, they like this
other one better, but that should come out in the evaluation
process through the competition process.

(0% Testimony Tr. at 27.

As discussed above, the primary reason that OIT wanted to purchase Cloverleaf
was the belief that it could meet the SEC’s backup and storage needs for less money than
other vendors such as EMC. However,” _ testified that assumptions about relative
costs should be tested through the process of competitive bidding:

Q: ... the preference for Dell in this example is based on
either assumptions about cost per unit or performance
criteria, where’s the line between going out and doing
research to determine that Dell is what I want [after
talking to other vendors]. Now I’ll go sole source
justification versus putting it out for the competition

30


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties, No redaction bas been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

technology or approach may have been determined, and
that will be presented as part of the pre-acquisition review.

(2% |Testimony Tr. at 21.

(‘explained that the PRB generally does not discuss a specific acquisition plan
or specific “product technology” until the project goes through pre-acquisition review.
°° ‘Testimony Tr. at 34.°° elaborated:

The pre-acquisition review is [a] process for the [project
manager] to present [to the PRB] the acquisition plan,
acquisition strategy, ... risk, risk management, [and]
mitigation ...

Id. at 34-35.

On April 14, 2008, the IOC approved the “concept” of the ADC Become{s]
Primary project. See April 14, 2008 IOC Meeting Minutes, attached as Exhibit 37, at 3-4.
However, the IOC was not presented with any information about Cloverleaf and there
was no discussion about any acquisitions OIT contemplated for that project. Jd. Without
any information other than general representations about the benefits of the project, the
IOC approved funding of $850,000 for the project. /d. On April 23, 2008, the project
was discussed at a PRB meeting. See April 23, 2008 PRB Meeting Minutes, attached as
Exhibit 38, at 5. The discussion included the fact that “PRB needs to determine ... if [a
pre-acquisition review] will be required.” Jd.!

On May 1, 2008, °°" OIT Director, approved certification of the
ADC Becomes Primary project as an “acquisition only investment.”'® See Exhibit 39.
°° ‘explained that such a certification waives the pre-acquisition review process. °""
Testimony Tr. at 39." _|also explained:

1s At its April 23, 2008 meeting, which was not attended by Walker andiX” \(see Exhibit 38 at 1),
the PRB discussed “lessons learned” from another project. See id. at 4-5. The OIG found that some of
these “lessons” applied equally to the Cloverleaf acquisition, including the following:

Often times “emergency” fix type projects are not as “immediate” as they
may initially seem and OIT should take the time to fully consider and plan
these projects.

OIT needs to work to foster a culture that encourages people to speak up
and address issues before they become major problems.

Id.at5.

‘6 On August 25, 2008, Walker also approved certification of the ADC Becomes Primary project as

an acquisition only investment. See Exhibit 40,

27


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

The OIG also interviewed the following three persons with relevant expertise
and/or knowledge of information pertinent to the investigation: (1) °°
ene OIT Finance and Administration; (2) °° ;
OIT Portfolio Support Branch; and (3) . , OIT Portfolio Support
Branch. See Excerpts of October 7, 2010 Interview Transcript (“OIT Finance and
Administration Interview Tr.”), attached as Exhibit 15.

Relevant Commission and Government Statutes, Regulations and Policies

I Competition and Contracting Act
41 U.S.C. § 253. Competition requirements.

(a) Procurement through full and open competitive procedures.

(1)Except as provided in subsections (b), (c), and (g) and except in the case
of procurement procedures otherwise expressly authorized by statute, an
executive agency in conducting a procurement for property or services--

(A) shall obtain full and open competition through the use of
competitive procedures in accordance with the requirements of this
title and the Federal Acquisition Regulation ...

(c) Use of noncompetitive procedures. An executive agency may use procedures
other than competitive procedures only when--
(1)the property or services needed by the executive agency are available from
only one responsible source and no other type of property or services
will satisfy the needs of the executive agency;

(f) Justification for use of noncompetitive procedure.

(1)Except as provided in paragraph (2), an executive agency may not award a
contract using procedures other than competitive procedures unless--

(A) the contracting officer for the contract justifies the use of such
procedures in writing and certifies the accuracy and completeness of
the justification;

(B) the justification is approved—


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

Scope of Investigation

The OIG reviewed approximately 1.2 million e-mails from thirteen custodians
from OIT and OAS. The OIG also reviewed numerous documents related to the
Cloverleaf acquisition provided by OIT, OAS, and various witnesses, including vendor
proposals, meeting minutes of the SEC’s Information Officers Council and OIT’s Project
Review Board, requisition requests, and purchase orders and supporting documentation.

Finally, the OIG conducted testimony on-the-record and under oath of the
following nine individuals:
eee OIT Specialist; taken on August 17, 20107”
Tr.”). Excerpts of Testimony Transcript attached as Exhibit 4.
2 OIT Specialist; taken on August 17, 2010 (“August 17,
190° fe) and September 22, 2010 (“September 22,
2010°%"° _Tr.”). Excerpts of Testimony Transcripts
attached as Exhibits 5 and 6, respectively.
3. ee OAS Management and Program Analyst; taken on August
18, 2010 °° |Tr.”). Excerpts of Testimony Transcript
attached as Exhibit 7.

4, Lewis Walker, OIT Deputy Director; taken on August 18, 2010 (“Walker
Testimony Tr.”). Excerpts of Testimony Transcript attached as Exhibit 8.

exTxcy

5. OIT Specialist; taken on August 19, 2010 (“August 19,
2010°"° Tr.”) and September 22, 2010 (“September 22,
2010 Tr.”). Excerpts of Testimony Transcripts attached

as Exhibits 9 and 10, respectively.

6 er OAS Contract Specialist; taken on August 19, 2010 ee
Testimony Tr.”). Excerpts of Testimony Transcript attached as Exhibit
11.

zee OAS Contract Specialist; taken on August 31, 2010
femme Tr.”). Excerpts of Testimony Transcript attached as
Exhibit 12.

3, oe on for Infrastructure Engineering; taken
on September 3, 2010°"° _Tr.”). Excerpts of Testimony
Transcript attached as Exhibit 13.

(XIN)

on September 9, 2010
Transcript attached as Exhibit 14.

3

__jtaken

ene _Tr.”). Excerpts of Testimony


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

Exhibit 27 at 1.

However, as discussed below, the cost estimates for Cloverleaf were incomplete
and Cloverleaf proved to be more expensive than other, better-known and less risky
alternatives.®

B. Proper Analysis and Testing of Cloverleaf Prior to its Acquisition Would
Have Demonstrated its Inadequacy at No Cost to the SEC

Instead of performing a technical analysis of Apple’s claims regarding Cloverleaf,
OIT made a cursory visit to another Cloverleaf client. See”° Testimony Tr. at 25-
26, 57-58. The genesis of that visit was a June 18, 2008 e-mail from” |to
and that stated:

I have reviewed the recently proposed solutions for backup
system at ADC and LA. I also reviewed the previous
proposals that was submitted by CSC back in Jan/Feb of
this year. The market survey report and technical approach _
of Apple/Cloveleaf [sic] seems attractive, however, °°
stated that he and” have not seen this solution in
action. So I decided that your proposal will be a go if and
only if the engineers can see these solutions implemented
somewhere. I know the procurement deadline is
approaching the scope of the project is large enough for us
to be cautious.

Exhibit 29 at 2.
8 Cloverleaf technology was completely new to the SEC. See August 19, 20 10% Tr.
at 35-36; (0X7 |Tr. at 32; August 17, 2010 Tr. at 11. In addition, within the

information technology industry Cloverleaf “was not [] widely known ... [i]t was an obscure thing.”

(XX) ___Tr, at 29-30. In fact, it was so obscure, that “if you did a Google search on it, you couldn’t find
it” Id. testified:

[Cloverleaf] wasn’t known back then when I first heard about it, and it’s
still not known. You know. As a matter of fact, I think they were going
out of business at this point, ...

August 17, 2010" 9 Testimony Tr. at 10. pmo

testified

Look at the industry and where they’re doing it. Other big agencies, other
big people that had this same problem, are any of them doing this? You
know, like, well, no and that most people I know when I’ve heard talking.
about colleagues out in other agencies about this, they laughed. They're
like, what are you guys doing over there?

(OxTKe)

August 19, 2010, Testimony Tr. at 22.

16


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

(1) Multiyear contracting, with a requirement in the solicitation that
a price be submitted only for the total multiyear quantity; or

(2) Priced options for additional quantities that, together with the
firm contract quantity, equal the program requirements.
48 C.F.R. pt. 3.501.

B. Full and Open Competition

The FAR “require[s], with certain limited exceptions, that contracting officers
shall promote and provide for full and open competition in soliciting offers and awarding
Government contracts.” See 48 C.F.R. pt. 6.3. Subpart 6.3 outlines exceptions to that
requirement as follows:

6.300 Scope of subpart.

This subpart prescribes policies and procedures, and identifies the statutory
authorities, for contracting without providing for full and open competition.

6.301 Policy

(a) ... Contracting without providing for full and open competition or full
and open competition after exclusion of sources is a violation of statute,
unless permitted by one of the exceptions in 6.302.

6.302-1 Only one responsible source and no other supplies or services will
satisfy agency requirements.

(a)(2) When the supplies or services required by the agency are available
from only one responsible source ..., and no other type of supplies or
services will satisfy agency requirements, full and open competition
need not be provided for.

(d)(1) Contracts awarded using this authority shall be supported by the
written justifications and approvals described in 6.303 and 6.304.


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

would have Apple not allowed that as far as you
know?

A: I think they probably would. They would have. I
think they, like any other vendor, they wanted to.
They had their foot in the door here at the SEC and
they were probably willing to do just about anything to
get that Cloverleaf project in so that they -- I mean --
that’s a lot of money for them to get their product out
into all the regions.

Q: Is there ... any reason why the system couldn’t have
been brought in and tested on a trial basis here at
headquarters OPC and ADC, as you were saying?

A: No. There’s no reason why it couldn’t have been.

(OxTKE) (NANCY

August 17, 2010) Testimony Tr. at 19-20, 22, 27-28, also testified that there
were no logistical obstacles to having tested Cloverleaf at the SEC before it was
purchased:

Q: [T]he testing could have been. It could have been
brought in and tested.

A: I think so... It’s not really a big deal for this size of
equipment. For this size of stuff, it should have been
easy, relatively easy to do.

(bX7)

‘(O |Testimony Tr. at 88.

Given that Cloverleaf was virtually unknown and untested, it was particularly
important for the SEC to have tested it on a no-cost basis. "explained the
importance of such a test in these circumstances:

[E]specially for a Cloverleaf type project where it’s new.
Nobody knows anything about it. You go out and you
can’t get any kind of rating information. You know. You
can’t get reviews. You can’t get any of that stuff. ... It
looks great on paper. It sounds great. I mean [the Apple
representatives make] it sounds like it can walk [on] water.
Okay. But you bring it in. You find out if it can really
walk on water.

August 17, 2010°"° _| Testimony Tr. at 24-25.°° also testified that since Cloverleaf
was a relative unknown in the information technology industry, it was particularly

22


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

REPORT OF INVESTIGATION

UNITED STATES SECURITIES AND EXCHANGE COMMISSION
OFFICE OF INSPECTOR GENERAL

Case No. OIG-523

Improprieties in the Selection of Information Technology
and the Award of a Sole-Source Contract

December 14, 2010


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

OPC. Hopefully any bugs with the installation will be
worked out in the lab exercise.

Exhibit 58 at 2. However, the “bugs” in the installation were not “worked out,” and
things quickly went downhill from there.

A. Cloverleaf Could Not be Installed in the LA Regional Office

OIT’s summary of Cloverleaf’s status in March 2009 included the following:

Oo

Id. By April 2009, OIT had abandoned any hope of installing Cloverleaf in the LA and
Denver offices and had the equipment shipped back to the OPC in Alexandria, Virginia.
OIT reported this development as follows:

April 30, 2009:

(exe)

Id, at.

However, those logistical problems paled in comparison to the performance
problems that arose when OIT began testing the Cloverleaf equipment.

B. Early Testing of Cloverleaf Proves it Unsuitable for the SEC

In June 2009, ‘summarized the results of the Cloverleaf tests in a memo for
OIT management that included the following:

%6 On March 24, 2009, OIT completed its security assessment of Cloverleaf. See Exhibit 59. That

assessment concluded that Cloverleaf “merits a HIGH RISK rating.” /d.

45


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

information systems, including criteria related to the
quantitatively expressed projected net, risk-adjusted
return on investment and specific quantitative and
qualitative criteria for comparing and prioritizing
alternative information systems investment projects;

(4) provide for identifying information systems
investments that would result in shared benefits or
costs for other Federal agencies or State or local
governments;

(5) provide for identifying for a proposed investment
quantifiable measurements for determining the net
benefits and risks of the investment; and

(6) provide the means for senior management personnel of
the executive agency to obtain timely information
regarding the progress of an investment in an
information system, including a system of milestones
for measuring progress, on an independently verifiable
basis, in terms of cost, capability of the system to meet
specified requirements, timeliness, and quality.

40 U.S.C. § 11312.

Results of the Investigation

I OIT’s Decision to Purchase Equipment Manufactured by Cloverleaf was
Flawed and Primarily Based on a Sales Pitch from a Vendor
In January 2008," a salesman from Apple, approached Lewis Walker,
OIT Deputy Director, about purchasing a virtual storage product manufactured by
Cloverleaf. Walker Testimony Tr. at 13. Walker recalled:

[T]he Apple Corporation is a vendor who markets us
regularly and they pitched the Cloverleaf product with
Apple storage behind it and said you really should look at
this. It’s a lot cheaper. It might be better, et cetera, et
cetera. ...

Id.

Shortly thereafter, Walker asked"
Infrastructure Engineering, to arrange for a presentation by Apple and Cloverleaf.
Testimony Tr. at 25. testified that the first time he heard of Cloverleaf was when

exrxey

9


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties, No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

(2) For a proposed contract over $650,000° but not exceeding $12.54
million, by the competition advocate for the procuring activity
designated pursuant to 6.501 or an official described in
paragraph (a)(3) or (a)(4) of this section. This authority is not
delegable.

48 C.F.R. pt. 6.3.
C. Exchange of Information
15.201 Exchanges with Industry before Receipt of Proposals.

(a)Exchanges of information among all interested parties ... are encouraged.
Any exchange of information must be consistent with procurement
integrity requirements. Interested parties include potential offerors, end
users, Government acquisition and supporting personnel, and others
involved in the conduct or outcome of the acquisition.

48 C.F.R. pt. 15.2.
Ill. Capital Planning and Investment Control (the Clinger-Cohen Act of 1996)

The Clinger-Cohen Act mandates, inter alia, that Federal Government executive
agencies “shall design and implement in the executive agency a process for maximizing
the value and assessing and managing the risks of the information technology
acquisitions of the executive agency.” 40 U.S.C. § 11312. Specifically, each executive
agency must have a Capital Planning and Investment Control (“CPIC”) process that shall:

(1) provide for the selection of information technology
investments to be made by the executive agency, the
management of such investments, and the evaluation
of the results of such investments;

(2) be integrated with the processes for making budget,
financial, and program management decisions within
the executive agency;

(3) include minimum criteria to be applied in considering
whether to undertake a particular investment in

3 As of 2008, this amount was $550,000. See Federal Register, August 30, 2010, Vol. 75, No. 167
at 53129-53135. Effective October 1, 2010, it was raised to $650,000. Id.

‘ As of 2008, this amount was $11.5 million. /d. Effective October 1, 2010, it was raised to $12.5
million. Id.


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties, No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

I’m just -- I have no idea why that was signed after the
fact. That’s not our normal process. That’s not good
processing. And I don’t have a reason.

Q: Okay. It’s not appropriate, is it?

A: No.

(exrxe)

| _Testimony Tr. at 83-84.25
Consequently, the OIG found that” placement of the three Cloverleaf
purchase orders on August 7, 2008, was a violation of the Competition and Contracting
Act and the Federal Acquisition Regulation.

D. OIT Improperly Helped Apple Craft its Proposal by Sharing Budget
Information and Apple Developed a Requisition Request that Fit Within
OIT’s Budget, but Did Not Include all of the Necessary Equipment for
the Cloverleaf Installation in Violation of the Federal Acquisition
Regulation

Section 15.201 of the FAR titled “Exchanges with Industry before Receipt of
Proposals” provides as follows:

(a)Exchanges of information among all interested parties ... are encouraged.
Any exchange of information must be consistent with procurement
integrity requirements. Interested parties include potential offerors, end
users, Government acquisition and supporting personnel, and others
involved in the conduct or outcome of the acquisition.

48 C.F.R. pt. 15.201 (emphasis added).

As discussed above, OIT decided to try out Cloverleaf using the $200,000 it had
budgeted i in fiscal year 2008 for “the Enterprise Tape Backup Expansion and Growth
project.” See Exhibit 22. According to a June 19, 2008 e-mail from” _|to Apple’s
salesman, vat some point before Apple submitted a quote, Apple had been

informed that OIT had budgeted exactly $200,000 for the project:

Can you update [the quote] since it has expired? Also, /
know we talked about keeping [it] at/under $200k, so if
that’s possible in the update, please do so.

3s PP Jatso acknowledged knowing that the purchase orders should not have been submitted

before the Justification and Approval was approved. (°° Testimony Tr. at 76.

41


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

that because of that violation, there was never any review of the technical merits related
to a $1 million acquisition of a technology that had never been used at the SEC from an
“obscure” '® company, Cloverleaf."

III. The Cloverleaf Acquisition Violated Several Provisions of the Competition
and Contracting Act and the Federal Acquisition Regulation

A. The Award of a Sole-Source Contract to Apple Violated the Competition
and Contracting Act and the Federal Acquisition Regulation.

The Federal Acquisition Regulation (“FAR”) governs the acquisition of goods
and services by the Federal Government. Typically, FAR requires any potential
acquisition to be subject to full and open competition. However, Subpart 6.3 outlines
some limited exceptions to the requirement. Specifically, Subpart 6.3 “prescribes
policies and procedures, and identifies the statutory authorities, for contracting without
providing for full and open competition.” 48 U.S.C. pt. 6.300. The exception that was
cited as justification for the sole-source Cloverleaf acquisition was Subpart 6.302-1. See
Justification and Approval for Other than Full and Open Competition for RQ 63000-08-
2455, 2470 and 2471 (“Justification and Approval”), attached as Exhibit 42, at 2. FAR
6.302-1 applies when there is [o]nly one responsible source and no other supplies or
services will satisfy agency requirements.”

acquisition review for the project. See Minutes of April 23, 2008 PRB meeting, attached as Exhibit 38, at
5.

® See Leta Testimony Tr. at 29-30. explained that OIT has a Technical Review Board
(“TRB”) that the PRB sometimes has review proposed hardware acquisitions “and considers questions of
how stuff can integrate, do we have any technical or security questions.” See OIT Finance and
Administration Interview Tr. at 17. The members of the TRB are “more ... technical, ... engineering kinds
of people” than the PRB. /d. at 20. The TRB addresses concems about how a pronosed hardware
acquisition may “integrate with our network.” Id. at 21, However, according to who often chairs
the PRB meetings, referrals of a proposed acquisition are “fairly uncommon.” /d. at 17, 22.

19 On January 13, 2009, {°° OIT Budget and Acquisition Specialist, sent Walker,
((bX7) ‘"q
©) and X10) Finance and Administration, an e-mail with the

subject “Clinger-Cohen Information — Does it apply to the SEC.” Exhibit 41." e-mail cited the
provisions of the Clinger-Cohen Act that clearly establish that the SEC is subject to the Act, and concluded
with the following:

What is troubling about this is that we go through this exercise almost
every six months and we need closure. Every time the rumor starts that we
aren’t subject to Clinger-Cohen it just makes the Portfolio Support Branch
job that much more difficult. We now require a management
“announcement” of the position so we can either continue to “try” and
comply with Clinger-Cohen or be assigned to positions that are appreciated
and valued.

(ex7xe)
Id.

29


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval,

Testing took place for six weeks in the OPC lab, ending on
16 April 2009.

Part of the reason for the testing taking six week, instead of
the originally estimated three, were difficulties with
integration of the [Cloverleaf] with the SAN fabric and
servers in the lab.

The Cloverleaf iSN will add complexity to the
administration of the SEC’s storage infrastructure.

When backups were run through [Cloverleaf], using
NetBackup, the backups took about 10% longer that when
the same backups were run without [Cloverleaf].

The SEC’s current Enforcement dataset is composed of
majority small files, and requires a server and storage
infrastructure that can perform at very high IOPS”’ ranges.
Unfortunately it appears that Cloverleaf iSN’s capabilities
and the SEC’s requirements are not well matched.

When the total cost of the Cloverleaff Promise solution
(e.g. iSN hardware, Cloverleaf software, storage, file
servers, SAN switch ports, etc.) is calculated, it ends up
being higher than most NAS solutions on the market. In
addition some NAS based solutions have 3rd party
verification of excellent small file performance.

The Promise storage arrays, purchased with the Cloverleaf,
should not be considered enterprise class storage. Their

” XO lexplained the primary performance problem with Cloverleaf as follows:

IOPS is 10's per second, how many things you can do in a second.
Throughput is how much water you can get down the pipe. And response
time is the time it would take for that water from Point A to Point B.
Those are the three, roughly, key metrics for storage.

(O© \Testimony Tr. at 27-28.


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

[A]n acquisition-only investment ... does not require a
preliminary acquisition review before the PRB, the project
review board. ... And I believe it also does not require
status reporting [to the PRB] on the procurement.

exTxcy

Testimony Tr. at 20-21.

The acquisition only investment certification “signifies OIT concurrence that [the]
investment poses no appreciable technical or delivery risk.” See Exhibits 39 and 40.
Both” and|”® testified that Cloverleaf did not satisfy the criteria for certification as
an acquisition only investment." Testimony Tr. at 33-34; September 22, 20101”
Testimony Tr. at 103-06. xplained:

[This thing is going to completely change the storage
architecture. ... [I]instead of just doing backups, it
completely rearchitects storage. So if it completely
rearchitects — it changes the whole game. It was a
significant change.

(®XTK)

Testimony Tr. at 32-33.

In fact, the first “condition[] for acquisition-only approval” on the certification
form approved by °””” on May 1, 2008, and by Walker on August 25, 2008, stated that
“{t]he investment product(s) do not require testing in the Configuration Management and
Quality Assurance Test Lab.” See Exhibits 39 and 40. According to" that
certification was false. See September 22, 2010" Testimony Tr. at 105-06. As
discussed below in Section IV, after OIT received the Cloverleaf equipment, it was tested
for six weeks in the OIT lab, and that testing revealed that Cloverleaf did not work for its
intended purpose.

The OIG found that OIT improperly certified Cloverleaf as an acquisition only
investment in order to avoid the mandatory pre-acquisition review.'” The OIG also found

" Apparently several individuals within OIT were responsible for the improper certification of the

ADC Becomes Primary project as acquisition only. As discussed above. the certification was approved
twice; once by” ‘and once by Walker. On both of those occasions,” was listed as the certifying,
official “per IOC.” See Exhibits 39 and 40. During his OIG testimony” testified that he did not recall
certifying the ADC Becomes Primary project as acquisition only. (°"” Testimony Tr. at 40, [°
suggested that some other OIT employee may have typed his name on the form without his knowledge. /d.
at 40-41, 0X09 an IT Specialist,in OIT’s Portfolio Support Branch, acknowledged that the
certification forms would have been prepared by an OIT employee within the Portfolio Support Branch.
See OIT Finance and Administration Interview Tr. at 27. No one could explain why the forms stated that
the certification was “per 10.” °° Testimony Tr. at 42; OIT Finance and Administration Interview
Tr.. at 28. In fact, when the IOC gave its “concept approval” to the project on April 14, 2008, there was no
discussion of waiving a pre-acquisition review. See Minutes of April 14, 2008 IOC meeting, attached as
Exhibit 37, at 3-4; OIT Finance and Administration Interview Tr. at 29. Moreover, according to the
minutes of an April 23, 2008 PRB meeting, the PRB had not decided as of that date whether to waive a pre-

28



This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

(i) in the case of a contract for an amount exceeding $500,000 (but
equal to or less than $10,000,000), by the competition advocate
for the procuring activity (without further delegation) or by an
official referred to in clause (ii) or (iii)

Il. Federal Acquisition Regulation

The Federal Acquisition Regulation (“FAR”) governs governs the acquisition of
goods and services by the Federal Government. See 48 C.F.R. pts, 1-99. Relevant
sections of the FAR are excerpted below.

A. “Buying-in”
Subpart 3.5, “Other Improper Business Practices,” of the Federal Acquisition

Regulation specifically addresses the practice of “submitting an offer below anticipated
costs” as follows:

3.501 Buying-in.
3.501-1 Definition.

“Buying-in,” as used in this section, means submitting an
offer below anticipated costs, expecting to—

(1) Increase the contract amount after award (e.g., through
unnecessary or excessively priced change orders); or

(2) Receive follow-on contracts at artificially high prices to recover
losses incurred on the buy-in contract.

3.501-2 General.

(a) Buying-in may decrease competition or result in poor
contract performance. The contracting officer must
take appropriate action to ensure buying-in losses are
not recovered by the contractor through the pricing
of--

(1) Change orders; or
(2) Follow-on contracts subject to cost analysis.
(b) The Government should minimize the opportunity for buying-in by

seeking a price commitment covering as much of the entire program
concerned as is practical by using--


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval,

Cloverleaf has asked the SEC to sign a Non Disclosure
agreement and an evaluation agreement. I have attached
the agreements for your review.

Exhibit 30. On May 6, 2008," _|reminded["” of his request, “I know you’ve
been very busy but I wanted to ask to see how it was going in approving our [proof of
concept] agreements with Cloverleaf and Data Domain.” See Exhibit 32.

On May 28, 2008," |e-mailed ("| the SEC’s non-disclosure agreement
and a revised Cloverleaf Product Evaluation Agreement and explained:

I’ve completed the demo agreement with Cloverleaf.
However, I made changes to their Agreement and I will not
sign, much less include, their NDA. If they want a Non-
Disclosure Agreement, we will use one of ours. Please
review the attached - do you want 30 days or 60 days?
They appear to normally provide it for 30 days but knowing
our environment, by the time we get it ready to use to
evaluate, 29 days will be gone! Please forward the attached
to your POC at Cloverleaf and see if they agree with the
revisions I had to make to their Agreement. The Federal
government follows the acquisition regulation and cannot
agree to certain terms.
Exhibit 33. Notwithstanding that! _“ and envisioned that the
equipment would be provided at no-cost for an evaluation, on the same day, May 28,
2008, Apple submitted to” [a discounted quote of exactly $200,000 for the purchase
of Cloverleaf equipment. See Exhibit 23.

(NTN)

During testimony in the OIG investigation,” _|discussed his April 29, 2008

e-mail to” |as follows:

I was looking to perform an evaluation of the product,

without being charged by the vendor to bring their

equipment in here to test the equipment out. ... 1 was

thinking that I could bring one or two pieces of equipment

in here, without deploying it to the production environment,

to get an understanding of how it would perform.
(XTC)

Testimony Tr. at 17.

However, for reasons unknown to" _| instead of trying Cloverleaf on a true,
proof of concept basis" decided to purchase $200,000 of Cloverleaf equipment and
install it at the SEC’s LA and Denver regional offices. See id. at 18, 47. ("

testified:


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

in conducting a procurement for property or services shall obtain full and open
competition through the use of competitive procedures.” 41 U.S.C. § 253; see also, 48
C.F.R. pt. 6.3 (“requir[ing], with certain limited exceptions, that contracting officers shall
promote and provide for full and open competition in soliciting offers and awarding
Government contracts”).

B. The Justification and Approval Was Not Reviewed by the SEC’s
Competition Advocate in Violation of the Competition and Contracting
Act and the Federal Acquisition Regulation

Section 253(f) of the CCA requires that the written justification for any contract
in excess of $550,000 that is awarded “using procedures other than competitive
procedures” must be approved “by the competition advocate for the procuring activity.”
41 U.S.C. § 253(f). See also 48 U.S.C. pt. 6.304 (the FAR requires that “the justification
for other than full and open competition shall be approved in writing [by the agency’s
competition advocate] [flor a proposed contract over $550,000 ...”). At the SEC, all
requests to sole-source an acquisition of more than $550,000 must be reviewed by"
(0% the SEC’s Competition Advocate. ["° ‘Testimony Tr. at 8. It is her
“responsibility to ensure that contract activity is competed to the maximum extent
practicable.” °° Testimony Tr. at 8.

The Justification and Approval for the Cloverleaf acquisition expressly applied to
all three of the Cloverleaf purchase orders. See Exhibit 42 at 1. Those three purchase
orders, in aggregate, were in excess of $550,000, and consequently should have been
reviewed by the SEC’s Competition Advocate. See!" — Testimony Tr. at 40 and 51.
7% explained:

Q: Who determines and how do they determine whether
or not [the acquisition is] going to be more than
550,000.

A: What they would look at is, you know, you pointed out
those RQ numbers at the top. ...The first place you
would look is those three RQ numbers, total them up.
And what’s the total value of those three? Because
those three are applicable to this J and A, and if those
three exceed $550,000, then this should have come
through me. So I would use that as your first
benchmark. ... But, again, it should specifically say in
[part 3 of the Justification and Approval] the total
estimated value. ...So that’s kind of odd that it’s
missing.

Q: Can I assume with reference to those three RQ
numbers that particularly if they were all dated the
same 7 day, I mean they were submitted at the same

36


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

time, that that’s what should have been done, is they
should have been totaled up and if the total exceeded
$550,000 you should have reviewed it?

A: Yes. And even if they weren’t issued on the same day,
the fact that they were referencing the three of them,
and all three of them applied at this acquisition, then
they should be considered as a whole.

Goo)

Testimony Tr. at 40-41.

and [°° admitted that they should have aggregated the three requisition
requests for purposes of the FAR. ”” |testified:

Q: Is it possible ... [the requisition requests] were broken
up to avoid [a] $500,000 threshold?

A: Well, now that you mention it -- but even then, even if
they had gamed the system to do that, when they came
to our office they should have been combined.

(mx7xe) ‘Testimony Tr. at 79. ("  |testified:

Q: ... And am I correct ...that the estimated value of
these three requisitions together was about $950,000?

A: Yes.

Q: And is it correct that from your earlier testimony that,
on that basis, °° the [competition]
advocate, should have reviewed this?

A: Yes,
Q: Okay. Do you know why that wasn’t done?

A: No, I don’t recall.

rene

Testimony Tr. at 73-74.

(RTE) (NTN)

The OIG found that and failure to submit the Justification and
Approval to the SEC’s Competition Advocate for review violated Section 253(f) of the
CCA and Section 6.3 of the FAR. Moreover, the award of the sole-source contract,
which in of itself was a violation of the CCA and the FAR, very likely would not have
occurred if the Justification and Approval had been reviewed by rene the SEC’s

Competition Advocate. See’ Testimony Tr. at 35,39. When” _ saw the

37


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval,

intention to expand Cloverleaf,” and that there is still a need to improve the SEC’s
backup system.” Jd. at 50-51,°""° testified that he anticipated that there would be a
competitive bid process in the future “to find out whether EMC, NetApp, Cloverleaf or
any [other] vendor” will be selected to solve the SEC’s backup problems. /d. at 51.

(view that the Cloverleaf acquisition was a success is not even shared by

Apple. On July 26, 2009, a representative from Apple asked Walker to serve as a
reference for the “Apple/Cloverleaf/Promise” system with another federal agency for the
award of a competitive contract. See Exhibit 64. Apple believed it was competing
against EMC and others. /d. at 1. Although requesting a reference, even the Apple
representative acknowledged that the SEC’s Cloverleaf acquisition had not yet resulted in
any benefits for the SEC:

Let me say in advance that I am deeply appreciate [sic] of
your time on this; and your willingness to assist —
especially given that you are not yet at the stage where you
are reaping the benefits of your Apple decision. But as I
said on our call last week -- and this is further reinforced by
my dealings with Cloverleaf and Promise over these past
few weeks -- you are in very good and capable hands.

Id, at 2.

© ‘concluded his testimony with this assessment of the Cloverleaf acquisition:

Q: Do you have anything you would like to add [or]
clarify?

A: No. I think it could have gone much differently and I
think we wouldn’t have wasted a million dollars. And
that’s the nutshell of it.

(exrxe)

Testimony Tr. at 120.

Conclusion

The OIG investigation found that the Cloverleaf acquisition violated several
provisions of the Competition and Contracting Act and the Federal Acquisition
Regulation. Specifically, the OIG found that OAS’s approval of the acquisition on a non-
competitive basis without sufficient justification violated Section 253(a) of the CCA and
Subpart 6.3 of the FAR. The fact that Cloverleaf acquisition was not reviewed and
approved by the SEC’s Competition Advocate was also a violation of Section 253(f) of
the CCA and Subpart 6.304(a) of the FAR. The OIG found that if the Cloverleaf
acquisition had been reviewed by the SEC’s Competition Advocate, it would likely not

52


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

installation at the SEC’s Operation Center (“OPC”) in Alexandria, Virginia, and the
SEC’s Alternate Data Center (“ADC”) in Ashburn, Virginia.

On June 30, 2008, OIT submitted the $200,000 Procurement Requisition
(“requisition request”) for the “Enterprise Tape Backup Expansion.” See Exhibit 24. On
June 18, 2008, twelve days before that requisition request," | e-mailed ° ]

I have all the market pricing (total $827) for the OPC/ADC
Cloverleaf solution. The total price includes Cloverleaf (8
iSNs), Apple storage (350 TB raw, 240.8TB usable), and a
new 48-port MDS switch for the ADC. I will give a project
overview to | today and prepare the [requisition
requests].

Exhibit 34,"

On July 8, 2008, one week after submitting the $200,000 requisition request for
the Cloverleaf backup project in the LA and Denver offices, OIT submitted two separate
requisition requests for $462,432.50 and $311,398.48 for the “ADC Becomes Primary”
project.'? See Exhibit 35.

I. The Cloverleaf Acquisition did not Undergo a Pre-Acquisition Review and
did not Satisfy the Criteria for an Exemption from that Review in Violation
of the SEC’s CPIC Procedures

The SEC has established a Capital Planning and Investment Control (“CPIC”)
process for the approval and oversight of Information Technology (“IT”) investment
projects. See March 2010 OIG Audit at 1. The primary purpose of the CPIC process is
to establish a strategic approach as to how the Commission uses its IT funds. Jd. The
SEC’s CPIC process is controlled by three governing boards: (1) the Information
Technology Capital Planning Committee (“ITCPC”); (2) the Information Officers
Council (“IOC”); and (3) the Project Review Board (“PRB”). /d. All projects must be
reviewed initially by the PRB and must then be approved by the IOC. Jd.

The IOC meets every month and is comprised of senior officers within the
Commission, including senior officials from divisions and offices other than OIT. Jd.

a Apple had submitted a quote to Walker on June 13, 2008, for some of this equipment. See Exhibit
36.
8 As discussed above, the Cloverleaf acquisition involved three separate requisition requests from
OIT; one dated June 30, 2008, and two dated July 8, 2008. None of the witnesses who testified in the OIG
investigation could recall or explain why the three requisition requests were submitted separately. See, e.g.,
XO) Testimony Tr. at 74-76;(% | Testimony Tr. at 49-50/° ‘Testimony Tr. at 48; September 22,
2010 _| Testimony Tr. at 72. Nor could any of the witnesses think of a legitimate reason for making
the Cloverleaf acquisition with three requisition requests. See, eg [Testimony Tr. at 74-76;
September 22, 2010 | Testimony Tr. at 74.

25


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

REPORT OF INVESTIGATION

UNITED STATES SECURITIES AND EXCHANGE COMMISSION
OFFICE OF INSPECTOR GENERAL

Case No. OIG-523

Improprieties in the Selection of Information Technology
and the Award of a Sole-Source Contract

Introduction and Summary of Results of Investigation

On September 24, 2009, the Securities and Exchange Commission’s (“SEC” or
the “Commission”) Office of Inspector General (“OIG”) opened an investigation into the
SEC Office of Information Technology’s (“OIT”) August 2008 acquisition of
approximately $1 million of certain equipment from Apple Inc. (“Apple”). The
equipment included some items manufactured by Apple and some virtual storage
equipment manufactured by Cloverleaf Communications Inc. (“Cloverleaf’).' The
purchase from Apple of Cloverleaf products and associated Apple products is hereinafter
referred to collectively as the “Cloverleaf acquisition.” The Cloverleaf acquisition was
one of four OIT projects reviewed by the OIG as part of its audit of the SEC’s Capital
Planning and Investment Control (“CPIC”) process. See Report No. 466, “Assessment of
the SEC Information Technology Investment Process,” March 26, 2010, (“March 2010
OIG Audit”), attached as Exhibit 1.2 Based on the findings of that audit with respect to
the Cloverleaf acquisition, the OIG decided to open this investigation.

} Cloverleaf was an Israeli company that marketed its Intelligent Storage Networking System
(“iSN”). See Exhibit 2. Cloverleaf was purchased by Dot Hill Systems Corp. in January 2010. See
Exhibit 3 at 1. As discussed below, in 2008, Apple was the only vendor licensed by Cloverleaf to sell its
technology to the federal government.

2 The March 2010 OIG audit found that, inter alia:

[S]everal program improvements are needed within the CPIC process
regarding the Commission's implementation of its CPIC policies and
procedures .... Specifically, we found that two out of four investments
we reviewed in a judgmentally-selected sample did not follow the
process prescribed in the CPIC policies and procedures and led to
significant decisions being made regarding IT investments without a
meaningful review by the appropriate boards,

See March 2010 OIG audit at iii.


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

We don’t step back and take the time to do the long-term
engineering understanding the total cost of ownership.

Like Cloverleaf is honestly a prime example of that. ...
They didn’t scope what the performance requirement. ...
So in Cloverleaf, they didn’t buy enough storage. They
didn’t buy any of that connectivity. We ended up having to
go buy, you know, buy ports on switches.

errs Testimony Tr. at 22-23.

Moreover, as discussed above, one week after OIT submitted its requisition
request for the Enterprise Tape Backup Expansion project, it submitted two additional
requisition requests totaling $773,000 for the ADC Becomes Primary project. By e-mail
dated July 25, 2008, Apple salesman” | put ) OAS Contract Specialist,
on notice that all three of the requisition requests were tied together. See Exhibit 57.
OO | e-mailed” __|in an effort to “try and track down the other two Cloverleaf /
Promise orders.” Id. °° stated that:

I’mstill a little worried that you only see the much smaller
Cloverleaf deal. That deal can only book if the other two
deals come with it. So, I’m a little worried, that is all.

Id. (emphasis added).

As a result of the several violations of the CCA and FAR, discussed above, OIT
and OAS circumvented the required competition process to purchase approximately $1
million of Apple and Cloverleaf equipment. Moreover, as discussed below, instead of
saving the SEC money, which was OIT’s purported reason for the acquisition, the
acquisition resulted in a significant waste of SEC resources.

IV. As Soon as OIT “Tried Out” Cloverleaf, it was Apparent that it was a
Failure

An OIT status report for Cloverleaf detailed a litany of problems with Cloverleaf.
See Exhibit 58. According to the report, the Cloverleaf equipment was received in
September 2008. /d. at 2. The change from CSC to Lockheed caused a five-month delay
in the effort to implement Cloverleaf. August 19,2010” Testimony Tr. at 47-49, In
February 2009, OIT summarized Cloverleaf’s status as, inter alia:

All hardware that was ordered last year has bee[n] received.
So far we have identified that two more Apple Xserves will
be need[ed] before installation takes place at the LA and
Denver sites. Currently the plan is to install the Cloverleaf
iSN system at the LA / Denver sites in April of 2009.
Starting March 10th, the installations for LA and Denver
will be replicated (as close as possible) in the lab at the

44


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

(exmxey (Pxrxe)

Exhibit 52 (emphasis added). r tified, “I do know knew to
keep it under [$200,000].” August v, 10 Testimony Tr. at 17. See also
August 7, 2008 e-mail from/TM |to!° _| attached as Exhibit 53 (referring to the
purchase of Cloverleaf equipment for the backup project as “the deal that (°°

needed me to match his budget of 200k”).

(ex7xe)

|acknowledged that it was improper for OIT to share information about its
budget before Apple submitted its proposal. "Testimony Tr. at 45-46. Walker
also acknowledged that it was improper for OIT to reveal its budget to a vendor in order
for the vendor to prepare a quote. Walker Testimony Tr. at 28-29.

Yet, after learning that OIT had a $200,000 budget for backup expansion, Apple
dutifully submitted a quote for exactly $200,000 to © bn May 28, 2008. See Exhibit
23. On August 7, 2008, e-mailed" asking for an explanation of the
$200,000 proposal since the supporting materials added up to either $196,803.82 or
$240,412.80. See Exhibit 53." responded and justified the $200,000 quote as
follows:

[W]e’ve waived several things on the 200k deal. [T]his is
the deal that °° _ _ineeded me to match his
budget of 200k. The normal discounted price was to be
240k, but he only had 200k. So we backed into his budget
based on the much larger order.

Id,

Thus, the OIG found that Apple used the information it improperly obtained from
the SEC to craft its offer in the exact amount that was budgeted by the SEC for the
contract.

In addition, Apple intentionally did not include all of the equipment that was
necessary to install Cloverleaf in order to submit a $200,000 quote, and planned on
having the SEC subsequently purchase that equipment, which was also a violation of the
FAR. See Exhibit 54. The FAR also does not allow a vendor to submit a proposal that
does not include all of the necessary items for the project and is below anticipated costs
with the expectation of subsequently increasing the amount or receiving follow-on
contracts. See 48 CFR pt. 3.501.

(eXTXC)

on August 20, 2008, two weeks after the SEC ordered Cloverleaf, e-

we fe
mai

If any additional end of the year funding comes available,
I'd like to try and get you to order two more xserve’s for
the storage project for the test. We went so skinny that I
didn’t include things that I would have normally included.

42


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

On February 1, 2008, four days after the first meeting with Cloverleaf, Apple
salesman" le-mailed © The [Cloverleaf solution is rather fantastic. Apple is a
big believer in this technology.” Exhibit 18,°'"° testified that he had believed
Cloverleaf was “promising,” based on the representations made by the Apple and
Cloverleaf salesmen. pr ‘Testimony Tr, at 26. However," acknowledged that OIT
should have conducted its own. engineering analysis of Cloverleaf instead of relying on
representations from salesmen. /d. But, as discussed below, that analysis was not done
before Walker and’””” decided to make the Cloverleaf acquisition.

((OXTXC) ((ox7XC)

OIT Specialist, described his perception of why Walker and
decided to make the Cloverleaf acquisition as follows:

[T]hey got the Cloverleaf because the Apple guy,
essentially, made contact with them and said, hey, I’ve got
this [great] product that will fix your issue. So I think they
bit it hook, line and sinker.

(ren7xey :
Testimony Tr. at 76.

A. OIT Decided to Use the $200,000 it had Budgeted for Addressing the
SEC’s Backup Problems to Try Out Cloverleaf without Proper
Consideration of Available Alternatives.

wnat early 2008, the SEC’s method of hacking up its data had become unworkable.
See Testimony Tr. at 41. OIT Specialist, described the problem as

follows:

8)

(exrKe)

Testimony Tr. at 17.

(rexmxey

of the Server and Storage Group, explained:

We back up -- our backup technology, rather, is a tape-
based system. We back up data to tape reels, and we store
the tapes for re-use. There is a large amount of data, and
we have problems completing the backups in a timely
fashion. We have to do a complete backup, for example,
on the weekends of all the data. And there can be so much
that the backup won’t complete in time. We have issues

12


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

As discussed above, Walker and” were eager to make the Cloverleaf
acquisition. Walker testified that he was not involved in deciding exactly how OIT
would try out the equipment purchased as part of that acquisition. See Walker Testimony
Tr. at 19.°° and Ja” contractor, decided that the backup problem
afforded an opportunity to do just that.” See August 17,2010(°° ‘Testimony Tr. at 61

°° noted that” and” “put most of [the Cloverleaf acquisition] together.”

©O ‘Testimony Tr. at 72-76)" |also recalled that when he subsequently asked"
why Cloverleaf had been purchased," _|told him, “Well, we were told to buy it,
okay, from upper management.” Testimony Tr. at 75-76.

pe explained his understanding of that decision as follows:

My understanding was that [Cloverleaf] would provide us a
way to ensure successful backups of data of our -- primarily
our litigation data [within the Enforcement Division],
which is a huge component of our storage requirements.

(exmxcy

‘Testimony Tr. at 13.”

OIT decided to try out Cloverleaf using the $200,000 it had budgeted in fiscal
year 2008 for “the Enterprise Tape Backup Expansion and Growth project.” See Exhibit
22. On May 28, 2008, Apple submitted a discounted quote for exactly $200,000 to!” |
See Exhibit 23. On June 26, 2008,” submitted the $200,000 Procurement
Requisition (“requisition request”) for the “Enterprise Tape Backup Expansion” to OAS.
See Exhibit 24.

frwrxer and” all testified that Cloverleaf was not the only vendor
ry (bx7KC) .
that offered a solution to the SEC’s backup problems. - also testified that Walker
was aware of that fact, Testimony Tr. at 32." _ testified:

Q: .... Did EMC have an option or an alternative that
would have at least at the time seemed to meet the

A(OX7NC) ]

7 On February 9, 2009," _ sent an e-mail from his personal account to his SEC account

stating:
wos . . (@XTKCY |,

This is probably the start of a number of emails to track the issues. _is
basically laying the blame for the backup situation on my fect and the fact
that Cloverleaf may not provide the answer. He seems to have forgotten
that he was led by" as well. Last Friday he told me in his
office that this was my problem and my job and I was in serious trouble. It
was more informational than threatening but it was obvious he was trying
to absolve himself of the issue. ... Obviously I can not [sic] rely onTM |

Exhibit 21.
14


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction
disclosure to third parties. No redaction has been performed by the Office of Inspector General.

before

Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

C. OAS Purchased Cloverleaf Before the Justification and Approval was
Approved in Violation of the Competition and Contracting Act and the

Federal Acquisition Regulation ..........ssssssssssessssessssessseesesesesssesssesssseess

D. OIT Improperly Helped Apple Craft its Proposal by Sharing Budget
Information and Apple Developed a Requisition Request that Fit Within
OIT’s Budget, but Did Not Include all of the Necessary Equipment for
the Cloverleaf Installation in Violation of the Federal Acquisition

Regulation.......ccscsessessssssssccsssssesssessssesssssssesescseessscsuscssscusensesaeecesenecenseseee

IV. As Soon as OIT “Tried Out” Cloverleaf, it was Apparent that it was a Failure.

A. Cloverleaf Could Not be Installed in the LA Regional Office.

B. Early Testing of Cloverleaf Proves it Unsuitable for the SEC.............06

C. Some of the Cloverleaf Equipment is Currently Being Used for a
Different Storage Purpose Than it Was Intended, but is not Performing
as Well as Less Expensive Alternatives. ..

CONCHUSION.......sescsessessseesssessesssesssesssesssessessessssssecssssssessessessseesssesseesuesssesesssenseeese



This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

(XTX) (exe)

Testimony Tr. at 49. also admitted that “the estimated value of [the] three
Tequisitions ... should have been there.” Testimony Tr. at 73.

Even OIT acknowledged that the Justification and Approval was inadequate to
justify a sole source contract. testified:

Q:_... [BJased on your experience in writing justifications
and approvals, and working with the Office of
Acquisitions on those documents, do you believe this
is an adequate justification for sole source contracting?

A: ... Based on my experience today, no.

(@xTx)

Testimony Tr. at $5-56. Walker testified:

Q: ...[W]hat [the Justification and Approval] [essentially]
says is that the SEC has determined it needs
Cloverleaf; that Apple has represented that it is the
only vendor that sells Cloverleaf [to the federal
government); that on-line market research was done to
confirm that there was no other vendor that the SEC
could acquire Cloverleaf from. [Based on] [yJour
understanding of the project, is that sufficient to justify
the sole source contract?

A: Well, based on the requirements of number 3, I mean,
again, strictly my belief, personal opinion is no. I
wouldn’t necessarily say it’s sufficient.

Q: [I]t seems to me like it begs the question. The question
is why is Cloverleaf the only solution for the need.
And what it says is the need is Cloverleaf. Well, of
course ... Apple is the only [vendor] that sells
Cloverleaf, but would you agree that the answer kind
of begs the question.

A: My opinion is I would have to agree with that.
Walker Testimony Tr. at 46-47.
The OIG found that the award of a non-competitive contract to Apple for the
Cloverleaf acquisition was a violation of the Federal Acquisition Regulation, and was in

contravention of the central tenet of the CCA and the FAR; that “except in the case of
procurement procedures otherwise expressly authorized by statute, an executive agency

35


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

important to have evaluated it on a no-cost basis. August 19, 2010/°° |Testimony Tr. at
14-15,

In light of Cloverleaf’s ultimate failure, discussed in detail below, to perform as
intended," testified:

Q: ... [mean looking back in hindsight, do you think that
it would have been a good idea to have brought it in?

A: Absolutely.

ere

August 17, 2010 Testimony Tr. at 27.

For no apparent reason, the no-cost, proof of concept test of Cloverleaf that was
originally planned became a $1 million'® acquisition to simply, as Walker testified, “try it
out.”'' As discussed below, the $1 million Cloverleaf ‘experiment was a failure from the
outset.

C. Key OIT Staff were not Consulted about the Decision to Acquire
Cloverleaf

Within” branch, some staff members are primarily responsible for the
SEC’s operational storage and other members of the group are focused on the SEC’s
backup issues. See August 17, 2010(°° ‘Testimony Tr. at 8-9{°° Testimony Tr. at
12-13. Because of the manner in which Cloverleaf was intended to backup the SEC’s
data, it created several significant issues for the SEC’s operational storage architecture.
©) ‘Testimony Tr. at 30, 32-33, 51; August 19, 2010 °°” Tr. at 13.
However, none of the OIT staff who worked in the group responsible for operational
storage were consulted before the Cloverleaf acquisition. See August 17, 2010 (°°

" As discussed below, OIT quickly added another $763,000 Cloverleaf acquisition for the SEC’s
Washington, DC office to the $200,000 it purchased for the LA and Denver offices,

" (XX defended the lack of pre-acquisition testing of Cloverleaf as follows:

Q: The Cloverleaf technology, prior to these acquisitions, had never been
tested or implemented anywhere in the Commission’s environment,
correct?

A: That is correct. It never had been tested. ... However, we rely on the
market research survey and we rely on the engineering design of my
staff to determine the feasibility and the requirements, whether the
product meet the requirement for these products. And when it
coming to testing, it will be comparison between the test result and
the vendor at the time of the product feature.

(XO Testimony Tr. at 56-57.

23


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

Table of Contents
Introduction and Summary of Results of Investigation ...........s.sssscssssessseessesssessseessecseeesee 1
Scope of Investigation
Relevant Commission and Government Statutes, Regulations and Policies... 4
I. Competition and Contracting Act ......c.sssessssessssessssssssessecsesnsenseatessessensssessseneessenees 4
Il. Federal Acquisition Regulation ...........sssssssssssssssssessssesssesssesacssvesssesssesnseseeesseenseeseenne 5
III. Capital Planning and Investment Control (the Clinger-Cohen Act of 1996).........0+ 8

Results of the Investigation ...

I. OIT’s Decision to Purchase Equipment Manufactured by Cloverleaf was
Flawed and Primarily Based on a Sales Pitch from a VendOl...........s.sssessseesseeeseseeees 9

A. OIT Decided to Use the $200,000 it had Budgeted for Addressing the
SEC’s Backup Problems to Try Out Cloverleaf without Proper
Consideration of Available Alternatives. ...........ssessssseees

B. Proper Analysis and Testing of Cloverleaf Prior to its Acquisition Would
Have Demonstrated its Inadequacy at No Cost to the SEC

C. Key OIT Staff were not Consulted about the Decision to Acquire
Cloverleaf .......c.csessecsessesssssesseesesscsscssvssesssesssesssssssssessssvesessssssssessessesseeseseeseeses 23

D. Before the Part of the Cloverleaf Acquisition that was Intended for Data
Backup in the LA and Denver Offices was Made, OIT Decided to
Purchase another $773,000 of Cloverleaf and Associated Apple
Equipment for a Different Purpose at SEC Headquarters..........csssesssssssseseseese 24

II]. The Cloverleaf Acquisition did not Undergo a Pre-Acquisition Review and
did not Satisfy the Criteria for an Exemption from that Review in Violation
Of the SEC’s CPIC Procedures .......sssscssssessssessesssssesssneessnecssseenssvesssecesseesssneessestenes 25

III. The Cloverleaf Acquisition Violated Several Provisions of the Competition
and Contracting Act and the Federal Acquisition Regulation...

A. The Award of a Sole-Source Contract to Apple Violated the Competition
and Contracting Act and the Federal Acquisition Regulation. ............s.sssess00 29

B. The Justification and Approval Was Not Reviewed by the SEC’s
Competition Advocate in Violation of the Competition and Contracting
Act and the Federal Acquisition Regulation..........cccssssssssessssssesesseneseseeneee 36


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties, No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

management has shown itself:to be complex and “buggy”

See Exhibit 60.

© | generally described his supervisors’ reaction to the test results as follows:

I just stepped back and went to management, you know,
with these problems and said, hey guys, you know, this
thing is not going to work or it’s highly unlikely it’s going
to work this way... They of course - - and" .
said, “You’re being negative, you know” or something like
that °° didn’t take it well. He was a little flustered by

it. By that time they’d already bought the equipment. It
had been delivered.

August 19, 2010° |Testimony Tr. at 87-88.” described one meeting in particular
where he and|*”° explained why Cloverleaf would not work for the SEC as follows:
poe was nervous, I get the sense. ... I laid out
in a document, because [there] were two parts. We were
basically saying, you know, this is what we see of the
Cloverleaf and this is the results we had of our market
research study on NAS. So the idea being I come into your
office as a manager. I want to tell you about a problem, but
I also want to tell you about [the] solution. I [don’t] want
to be the guy that comes in with [just] problems. So we
wanted to say here’s the bad news and here’s the good
news, so we didn’t really get to the good news, because
[folded the big, large piece of paper over, and said,
“This information doesn’t leave this room.” Then he goes
out. ... So that ... kind of surprise[d] me. I wasn’t
expecting that reaction. I was expecting [him] to be mad,
but not to, you know, in essence say don’t say anything to
anybody about this, and run out the door.
Id. at 89-90." |testified similarly regarding OIT management’s reaction to the bad
news about Cloverleaf:
And what we got back [from | Walker and oo
(xx) |was, well, you guys are just trying to, you know,
can Cloverleaf. It’s you’re not being objective. You’re
being anti-Apple, anti-Cloverleaf... And it was weird.
When we wanted to meet with [Walker] ... [he] wouldn’t
do it without Apple being present. And it just seemed

47


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

process and letting those vendors formally provide bids
with the cost and performance issues.

A: You make a very valid point, and a lot of times,
sometimes, it could be judgmental, but I do agree with
you that the cost differential is best tested in the
competitive market.

(or Testimony Tr. at 24-25.

10 | who was the individual in OAS that worked closely with OIT in
connection with the Cloverleaf Justification and Approval, testified that he understood
OIT’s stated need in this matter was for a “a system of storage management and disk-
based backup and recovery.”2° °° Testimony Tr. at 64." _ agreed that a belief
that Apple could provide less expensive storage was not a proper basis for requesting a
sole-source acquisition:

Q: ... [I]s, an assumption about cost and an assumption
that the Apple product was cheaper, is that sufficient
basis to justify going out with a sole source contract to
Apple?

A: No, not just based solely on cost.

Q: Isn’t that exactly the type of situation that’s supposed
to go out for competitive bid to test who does have
either the cheapest or the best value?

A: From pure cost standpoint that would be a point of
competition.

20 Gu |prematurely decided to support OIT’s request for a sole-source contract at the very

beginning of the process. See Exhibit 43. On July 16, 2008, °° | e-mailed |" | regarding requisition
request 2455 the following:

I received the above [requisition request], but I didn’t see a noncompetitive
justification for Cloverleaf and Promise. Neither of these two are available
from Apple’s GSA Schedule and are considered as open market purchases.
Can you provide the justification document and I'll support it.

Id. at 2 (emphasis added). On July 20, 2008," | e-mailed!" |a sample Justification and Approval
and stated:

1 would think that you basically want to say why you need Cloverleaf and
Promise (and nothing else will do) and that these two products are only
available from Apple. J will support it.

Id. at | (emphasis added).

31


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

understand correctly that the purchase orders should
have come after the justification?

A: Yes.

(exTxe) exryey

Testimony Tr. at 53. explained:

[Y]Jou’re required by the FAR to justify when you’re not
using full and open competition. Then that vehicle is this J
and A. ...If you go ahead and just award a sole source
contract without the approval of that particular strategy,
you’ve not allowed for CCA, which is “Competition and
Contracting Act.” Okay. That says that you must compete
to the maximum extent practical. ...So if you don’t have
the J and A, which is your authority to not seek full and
open competition before you award, to me you’re not
violating CCA, but, you know, it brings into question did
you really comply with CCA.

eo Testimony Tr. at 55.
9 [acknowledged knowing that the purchase orders should not be signed
before the Justification and Approval was approved, but did not have an explanation of
why she had done just that:

Q: ...You notice the date on the [purchase orders is]
August 7, 2008. ... [T]he justification and approval for
these, is dated August 11th, 4 days later.

A: Oh, boy. On both signatures?
Q: Yes.

Oh, my word. [Justification and Approval] was
approved after they were signed.

Q: [The purchase orders] obligated the funds, is that
correct?

A: Yes. ... Oh, I’m -- I have no idea why that was signed
after the fact. That is not our process. And I have no
recollection of the circumstances that went behind that.
I was thinking -- you pointed out the fact that the value
was missing, and I was thinking, well, if they summed
it all up, that wouldn’t be a good thing for them. Wow.

40


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

See also, Walker Testimony Tr. at 8-12; Testimony Tr. at 14-19. The PRB’s
primary role is to select and evaluate IT investments that meet the strategic direction of
the agency and to provide sound and diverse advice to the Chief Information Officer
(“CIO”) regarding the Commission’s IT portfolio. See March 2010 OIG Audit at 2. The
IOC is responsible for periodically reviewing the results of completed investments. /d.
The IOC is also responsible for conducting periodic reviews of the entire IT portfolio. /d.

The PRB meets weekly and is charged with ensuring that IT investments are
selected, controlled, and evaluated after completion. /d. The PRB is chaired by the
Deputy CIO, Walker, and its members include all of the OIT Assistant Directors. Walker
Testimony Tr. at 9;°"° Tr. at 19. The PRB’s responsibilities include:

PRB will ensure program, project, or initiative plans
brought before the Board have been fully reviewed and
accurately describe the scope/requirements, life-cycle costs,
schedule, and risks. The PRB will identify adequate
resources prior to selection including resources to
appropriately mitigate identified risks, before
recommending approval by the CIO or IOC.

See PRB Charter, attached as Exhibit 65, at 2.

The SEC’s CPIC procedures mandate that any IT project in excess of $25,000
undergo a pre-acquisition review (“PAR”).'* See OIT’s CPIC Overview, attached hereto
as Exhibit 66. The pre-acquisition review and approval are conducted by the PRB unless
otherwise specified by the IOC or ITCPC. Id." _ explained the significance of a
pre-acquisition review as follows:

[A pre-acquisition review] is a second appearance before
the project review board, where an acquisition plan is
presented for their review and approval. ... A specific

wo pmo O1»reno for Finance and Administration, is a member of the PRB
and “often chair[s] the board.” See OIT Finance and Administration Interview Tr. at 17,
acknowledged, in general “discrepancies” between OIT’s CPIC policies and actual practices as follows:

And we're going to give you the way the process should work and then
be aware of two things. One is, as you well know being in the IG shop,
things don’t always work perfectly -- that there are pragmatic pieces and
we'll try to address those where you've got questions. The other, be
aware we have had a couple of IG audits of the CPIC processes and we
currently have -- the agency has recently retained Mitre Corporation to
review the CPIC processes. Because we are aware that there are some
inconsistencies. There are some discrepancies between what the policy
says, what some of the forms require, or may appear to require -- and
some of the things we're doing.

Id. at 5.

26


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

Walker told him that it could “address our storage and backup problem,” and asked him
to schedule a meeting with Apple and Cloverleaf. Id. On January 28, 2008, a
representative of Cloverleaf met with Walker{° and :

PO for the Server and Storage Group. See Exhibit 16. The Cloverleaf representative
summarized the meeting in an e-mail to the attendees dated January 29, 200-8, as
follows:

Thank you again for your time yesterday afternoon and
sharing your storage challenges with me. I am very
confident that we can save you money both in acquisition
and operation costs while providing a more flexible and
highly available environment. ...

Our “any-to-any” virtualization technology, when

combined with Apple’s low cost storage allows you to gain

more capacity and services without any sacrifice to

performance or reliability — an extremely compelling value
_ proposition in today’s cost-sensistive [sic] world. ...

... As we agreed, the next step is either for us to provide
you with an evaluation system installed in your lab or a
two-day visit to our corporate offices in Woodbury, NY to
have some hands-on time in our lab. A precursor to that
would be a follow up meeting face-to-face with our
Director of Professional Services to go over your
environment and applications in more detail. ...

I have also attached a white paper that outlines the testing
with we {sic] for the USAF in Europe using Exchange
2003. Apple and Cloverleaf were selected as a “technically
superior solution” at a “significant costs savings” when
compared to a similar EMC solution.

Id.
Walker described his view of Cloverleaf after the meeting as follows:

My understanding of it at the time I was involved, looking,
you know, talking about Cloverleaf or storage
virtualization, was that we should probably get some and
try it out. But what the decisions made as to what it was
purchased for and how it was used, I was not involved in
that that I recall.

Walker Testimony Tr. at 19 (emphasis added). Specifically, Walker explained he wanted
to try Cloverleaf out because he wanted to find a cheaper product for data storage:

10


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

really odd that why would you not meet with your own
folks without an outside vendor present.

°° 'Testimony Tr. at 44-45," [also testified that” reaction was to “hide the
mistake” of purchasing Cloverleaf:
. (oy7XC)
Q: ... I’ve heard of a meeting that was at where
problems with Cloverleaf were discussed. ... Well, he
said like, “This doesn’t leave the room.”

A: But yeah, there were some meetings where that didn’t
leave the room. ...

Q: So that instruction was intended to hide the mistake;
am I correct?

A: It would seem so.

(BxTKCY

Testimony Tr. at 38-39.2°""" testified that “this sentiment from some of [his]
supervisors ...not to share ... outside a certain circle the problems” came from Walker
and" not” Pd, at 117.

rexmncy (mxmxey

After receiving the results of the testing and” _| analysis, sent
“write-up” on August 13, 2009, that discussed two options for using the Cloverleaf

% XO testified that he did not recall having been in a meeting with OIT staff where he was “told

about problems with Cloverleaf, including the I/O problem, that it was too slow for enforcement data and
saying, in words or substance, this information doesn’t leave the room.” (2X Testimony Tr. at 62-63.

2» Oo)

On February 13, 2009, a representative from EMC e-mailed and(2” about Cloverleaf’s

compatibility with EMC equipment as follows:

We have researched your request related to if Cloverleaf has been tested or

approved through the EMC Interoperability Lab. Thus far, no one from the

lab or other EMC resources is familiar with Cloverleaf ... We can’t verify

how their systems will work or integrate with the EMC systems at this

time. We rely heavily on our lab to test and approve what is compatible

with our solutions for good reason. If Cloverleaf has not been EMC lab

certified then we can’t validate it with our systems.
Exhibit 61° responded with an e-mail to{2” stating, “Do not share other than with me.” Id.
(&® explained his understanding of 1 response as follows:

I think|®” jis trying ... to protect -- not protect us, but not -- you know.
it’s like just more bad news. ... I think he is more trying to help. ... fewer
does a real good job or tries to get things done on an even keel. Just a
really good guy to work for.

(x7) .
() Testimony Tr. at 118-19.

48


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General.
Recipients of this report should not disseminate or copy it without the Inspector Gencral’s approval.

C. Some of the Cloverleaf Equipment is Currently Being Used for a
Different Storage Purpose Than it Was Intended, but is not Performing
as Well as Less Expensive Alternatives.

After Cloverleaf was proven a failure in the OPC lab, ee and” attempted to
find some use for the equipment. August 19, 2010°”° Testimony Tr. at 92-95; °°
Testimony Tr. at 46-47. As a result of their efforts, OIT is making use of some, but not
all, of the Cloverleaf equipment for backing up some data at the ADC. 31/7 explained:

In Alexandria. The OPC side of the Cloverleaf does
nothing. It just sits there and does nothing because there is
no storage behind it. And then the gear that was slated for
the regions that is still sitting in our test lab, it would have
cost us, I think, [$40,000] more to [use it], but it won’t
really give us anything.

Pen Testimony Tr. at 49.>* However, the actual Cloverleaf equipment is actually

unnecessary for its current function of backing up data to the Promise storage arrays and

actually slows the process down when it is used as” >xplained:

And then what we tried to do is come up with [an
alternative use for Cloverleaf], ... we did a test of backup
performance using ... Cloverleaf [as] a backup target in
front of the Promise arrays .... As it came back, the test
was faster if we went straight to Promise. So I actually
[recommended taking] Cloverleaf out of the middle. But
because we bought it, we had to use it. .. It was a 10
percent [performance] penalty ... to [use] Cloverleaf.

Id. at 46-47 (emphasis added).

Moreover, OIT could have purchased significantly better and less expensive
hardware to serve the same purpose for which Cloverleaf is currently being used.” |
testified that, “By going through Cloverleaf, we end up adding complexity and cost. For
no[], no benefit.” Jd. at 54-55.°"° [explained further:

Q:_... [W]hat Cloverleaf is actually doing, if that had been
the original intent, say, well, we just want to have a
backup target for the subset of data that it’s backing up

a (XX) "| testified that the Cloverleaf / Promise storage arrays were providing only “a small part” of

the backup function at SEC headquarters. August 17, 2010 (°K Testimony Tr. at 42.
2 On July 7, 2010, another sole-source contract was awarded to Cloverleaf’s successor for
$107,004.28 in order to “support” the Cloverleaf system that was purchased in 2008. Sce various contract
documents, attached hereto as Exhibits 3 and 63.

50


This document is subject to the provisions of the Privacy Act of 1974, and may require redaction before
disclosure to third parties. No redaction has been performed by the Office of Inspector General,
Recipients of this report should not disseminate or copy it without the Inspector General’s approval.

with equipment that may fail during the backup operation,
which puts the backup at risk. And we were looking for a
way to ensure that we had successful backups, and we were
looking at a disk-based system to provide this capability to
ensure reliable backups.

©° Testimony Tr. at 13-14.

(XS) (OXTNC)

(XTC)

On February 8, 2008, ten days after the first meeting with Cloverleaf,
responded to” e-mail about approaches to the “[bJackup [sJituation within SEC.”
Id. at 1-2, responded to|""”"

(exrNey

Thanks, Don’t forget about the Cloverleaf. It has
potential of virtualization of the storage. I particularly like
the feature of taking a snapshot of the volume. It is
basically backing up to disk but not using the Netbackup
software.

Id. at 1.

On February 20, 2008, the Cloverleaf representative from the January 28, 2008
meeting e-mailed ene

I am following up with you to see if you had had the
opportunity to talk with your team about evaluating our
solution. During our meeting I offered to host you in our
Woodbury office or bring an evaluation system into your
lab. Can we have a short phone call this week to talk about
next steps?

Exhibit 20," replied, “I and" are very much interested in evaluating your
product. ... Can we do that next week?” /d.

13