<PAGE>
 
                                                                   EXHIBIT 10.29


 
[LOGO OF SILICON VALLEY BANK]

                          Amendment to Loan Agreement

Borrower:   Vitesse Semiconductor Corporation
Address:    741 Calle Plano
            Camarillo, California 93012

Date:       January 22, 1997

THIS AMENDMENT TO LOAN AGREEMENT is entered into between SILICON VALLEY BANK
("Silicon") and the borrower named above (the "Borrower").

   The Parties agree to amend the Loan and Security Agreement between them dated
May 8, 1991, as amended by that Amendment to Loan Agreement dated May 8, 1991,
as amended by that Amendment to Loan Agreement dated August 12, 1992, as amended
by that Amendment to Loan Agreement dated December 28, 1992, as amended by that
Amendment to Loan Agreement dated June 14, 1993, as amended by that Amendment to
Loan Agreement dated December 10, 1993, as amended by that Amendment to Loan
Agreement dated March 23, 1994, as amended by that Amendment to Loan Agreement
dated December 13, 1994, and as amended by that Amendment to Loan Agreement
dated January 2, 1996 (the "Loan Agreement"; unless otherwise defined herein,
terms defined in the Loan Agreement are used herein as therein defined), as
follows, effective as of the date of hereof:

   1.     Deletion of Sections 1A and 1B. Sections 1A and 1B of the Loan
Agreement are hereby deleted in their entirety.

   2.     Modification to Section 2.2 of Loan Agreement. Section 2.2 of the Loan
Agreement is hereby deleted and replaced with the following Section 2.2 and
Section 2.2A:

          2.2 Termination of Security Interest; Definition of Collateral. The
          Borrower's grant of a security interest in the Collateral under the
          original Loan Agreement is hereby deemed terminated and of no force
          and effect; and all references to the security interest or lien of
          Silicon in the Collateral as set forth in the Loan Agreement and
          related documents are deemed to be of no further force or effect.
          However, for definitional purposes only under this Agreement, the term
          "Collateral" as used herein shall mean all of the Borrower's interest
          in the types of property described below, whether now owned or
          hereafter acquired, and wherever located: (a) All accounts, contract
          rights, chattel paper, letters of credit, documents, securities,
          money, and instruments, and all other obligations now or in the future
          owing to the Borrower; (b) All inventory, goods, merchandise,
          materials, raw materials, work in process, finished goods, farm
          products, advertising, packaging and shipping materials, supplies, and
          all other tangible personal property which is held for sale or lease
          or furnished under contracts of service or consumed in the Borrower's
          business, and all warehouse receipts and other documents; and (c) All
          equipment, including without limitation all machinery, fixtures,

                                      -1-
<PAGE>
 
          trade fixtures, vehicles, furnishings, furniture, materials, tools,
          machine tools, office equipment, computers and peripheral devices,
          appliances, apparatus, parts, dies, and jigs; (d) All general
          intangibles including, but not limited to, deposit accounts, goodwill,
          names, trade names, trademarks and the goodwill of the business
          symbolized thereby, trade secrets, drawings, blueprints, customer
          lists, patents, patent applications, copyrights, security deposits,
          loan commitment fees, federal, state and local tax refunds and claims,
          all rights in all litigation presently or hereafter pending for any
          cause or claim (whether in contract, tort or otherwise), and all
          judgments now or hereafter arising therefrom, all claims of Borrower
          against Silicon, all rights to purchase or sell real or personal
          property, all rights as a licensor or licensee of any kind, all
          royalties, licenses, processes, telephone numbers, proprietary
          information, purchase orders, and all insurance policies and claims
          (including without limitation credit, liability, property and other
          insurance), and all other rights, privileges and franchises of every
          kind; (e) All books and records, whether stored on computers or
          otherwise maintained; and (f) All substitutions, additions and
          accessions to any of the foregoing, and all products, proceeds and
          insurance proceeds of the foregoing, and all guaranties of and
          security for the foregoing; and all books and records relating to any
          of the foregoing."

     3.   Schedule to Loan Agreement. The Schedule to Loan and Security
Agreement is hereby deleted and replaced with the Schedule to Loan and Security
Agreement attached hereto.

     4.   Financial Conditions and Statements. Section 3.7 of the Loan Agreement
is hereby deleted and replaced with the following:

          3.7 Financial Condition and Statements. All financial statements now
     or in the future delivered to Silicon have been, and will be, prepared in
     conformity with generally accepted accounting principles and now and in the
     future will completely and accurately reflect the financial condition of
     the Borrower, at the times and for the periods therein stated. Since the
     last date covered by any such statement, there has been no material adverse
     change in the financial condition or business of the Borrower. The Borrower
     is now and will continue to be solvent. The Borrower will provide Silicon:
     (i) Within 5 days after the earlier of the date Borrower's 10-Q report is
     filed or is required to be filed with the Securities Exchange Commission,
     such 10-Q report, a quarterly financial statement prepared by the Borrower,
     and a Compliance Certificate in such form as Silicon shall reasonably
     specify, signed by the Chief Financial Officer of the Borrower, certifying
     that at the end of such quarter the Borrower was in full compliance with
     all of the terms and conditions of this Agreement, and setting forth
     calculations showing compliance with the financial covenants set forth on
     the Schedule and such other information as Silicon shall reasonably request
     (the "Compliance Certificate") for the quarter then ended; and (ii) within
     5 days after the earlier of the date Borrower's 10-K report is filed or is
     required to be filed with the Securities Exchange Commission, such 10-K
     report, and complete annual financial statements, certified by independent
     certified public accountants acceptable to Silicon.

     5.   Fee. Borrower shall concurrently herewith pay to Silicon a fee in the
amount of [*], which shall be in addition to all interest and to all other
amounts payable hereunder, and which shall not be refundable.

                         [* Confidential information]

                                      -2-
<PAGE>
 
     6.  Representations True. The Borrower represents and warrants to Silicon
that all representations and warranties set forth in the Loan Agreement, as
amended hereby, are true and correct.

     7.  General Provisions. This Amendment, the Loan Agreement, any prior
written amendments to the Loan Agreement signed by Silicon and the Borrower, and
the other written documents and agreements between Silicon and the Borrower set
forth in full all of the representations and agreements of the parties with
respect to the subject matter hereof and supersede all prior discussions,
representations, agreements and understandings between the parties with respect
to the subject hereof. Except as herein expressly amended, all of the terms and
provisions of the Loan Agreement, and all other documents and agreements between
Silicon and the Borrower shall continue in full force and effect and the same
are hereby ratified and confirmed.

  Borrower:                              Silicon:

  VITESSE SEMICONDUCTOR                  SILICON VALLEY BANK
  CORPORATION

  By /s/ [ILLEGIBLE SIGNATURE]           By /s/ [ILLEGIBLE SIGNATURE]
    ------------------------------         ------------------------------
           Vice President                Title      Vice President
                                              ---------------------------

  By /s/ [ILLEGIBLE SIGNATURE]     
    ------------------------------ 
          Ass't Secretary

                                      -3-
<PAGE>
 
[LOGO OF SILICON VALLEY BANK]

                                  Schedule to

                                 Loan Agreement

Borrower:  Vitesse Semiconductor Corporation
Address:   741 Calle Plano
           Camarillo, California 93012

Date:      January 22, 1997

Credit Limit        An amount not to exceed $12,500,000.
(Section 1.1):

Letter of Credit    Silicon, in its reasonable discretion, will from time to
Sublimit            time during the term of this Agreement issue letters of
                    credit for the account of the Borrower ("Letters of
                    Credit"), in an aggregate amount at any one time outstanding
                    not to exceed $1,000,000, upon the request of the Borrower,
                    provided that, on the date the Letters of Credit are to be
                    issued, Borrower has available to it Loans in an amount
                    equal to or greater than the face amount of the Letters of
                    Credit to be issued. Prior to the issuance of any Letters of
                    Credit, Borrower shall execute and deliver to Silicon
                    Applications for Letters of Credit and such other
                    documentation as Silicon shall specify (the "Letter of
                    Credit Documentation"). Fees for the Letters of Credit shall
                    be as provided in the Letter of Credit Documentation.
                    Letters of Credit may have a maturity date up to twelve
                    months beyond the Maturity Date in effect from time to time,
                    provided that if on the Maturity Date, or on any earlier
                    effective date of termination, there are any outstanding
                    Letters of Credit issued by Silicon or issued by another
                    institution based upon an application, guarantee, indemnity
                    or similar agreement on the part of Silicon, then on such
                    date, Borrower shall provide to Silicon cash collateral in
                    an amount equal to the face amount of all such Letters of
                    Credit plus all interest, fees and costs due or to become
                    due in connection therewith, to secure all of the
                    Obligations relating to said Letters of Credit, pursuant to
                    Silicon's then standard form cash pledge agreement.

                    The Credit Limit set forth above and the Loans available
                    under this Agreement at any time shall be reduced by the
                    face amount of Letters of Credit from time to time
                    outstanding.

                                      -1-
<PAGE>
 
Foreign Exchange  
Contract Sublimit  Up to $1,000,000 (the "Contract Limit") may be utilized for
                   spot and future foreign exchange contracts (the "Exchange
                   Contracts"). The Credit Limit available at any time shall be
                   reduced by the following amounts (the "Foreign Exchange
                   Reserve") on each day (the "Determination Date"): (i) on all
                   outstanding Exchange Contracts on which delivery is to be
                   effected or settlement allowed more than two business days
                   from the Determination Date, 20% of the gross amount of the
                   Exchange Contracts; plus (ii) on all outstanding Exchange
                   Contracts on which delivery is to be effected or settlement
                   allowed within two business days after the Determination
                   Date, 100% of the gross amount of the Exchange Contracts. In
                   lieu of the Foreign Exchange Reserve for 100% of the gross
                   amount of any Exchange Contract, the Borrower may request
                   that Silicon debit the Borrower's bank account with Silicon
                   for such amount, provided Borrower has immediately available
                   funds in such amount in its bank account.

                   Silicon may, in its discretion, terminate the Exchange
                   Contracts at any time (a) that an Event of Default occurs or
                   (b) that there is not sufficient availability under the
                   Credit Limit and Borrower does not have available funds in
                   its bank account to satisfy the Foreign Exchange Reserve. If
                   either Silicon or Borrower terminates the Exchange Contracts,
                   and without limitation of the FX Indemnity Provisions (as
                   referred to below), Borrower agrees to reimburse Silicon for
                   any and all fees, costs and expenses relating thereto or
                   arising in connection therewith.

                   Borrower shall not permit the total gross amount of all
                   Exchange Contracts on which delivery is to be effected and
                   settlement allowed in any two business day period to be more
                   than $500,000 (the "Settlement Limit"), nor shall Borrower
                   permit the total gross amount of all Exchange Contracts to
                   which Borrower is a party, outstanding at any one time, to
                   exceed the Contract Limit.

                   Notwithstanding the above, however, the amount which may be
                   settled in any two (2) business day period may, in Silicon's
                   sole discretion, be increased above the Settlement Limit up
                   to, but in no event to exceed, the amount of the Contract
                   Limit (the "Discretionary Settlement Amount") under either of
                   the following circumstances (the "Discretionary Settlement
                   Circumstances"):

                     (i) if there is sufficient availability under the Credit
                     Limit in the amount of the Foreign Exchange Reserve as of
                     each Determination Date, provided that Silicon in advance
                     shall reserve the full amount of the Foreign Exchange
                     Reserve against the Credit Limit; or

                     (ii) if there is insufficient availability under the Credit
                     Limit as to settlements within any two (2) business day
                     period if

                                      -2-
<PAGE>
 
                     Silicon is able to: (A) verify good funds overseas prior to
                     crediting Borrower's deposit account with Silicon (in the
                     case of Borrower's sale of foreign currency); or (B) debit
                     Borrower's deposit account with Silicon prior to delivering
                     foreign currency overseas (in the case of Borrower's
                     purchase of foreign currency);

                  Provided that it is expressly understood that Silicon's
                  --------
                  willingness to adopt the Discretionary Settlement Amount is a
                  matter of Silicon's sole discretion and the existence of the
                  Discretionary Settlement Circumstances in no way means or
                  implies that Silicon shall be obligated to permit the Borrower
                  to exceed the Settlement Limit in any two business day period.

                  In the case of Borrower's purchase of foreign currency,
                  Borrower in advance shall instruct Silicon upon settlement
                  either to treat the settlement amount as an advance under the
                  Credit Limit regarding the Revolving Loan Facility, or to
                  debit Borrower's account for the amount settled.

                  The Borrower shall execute all standard form applications and
                  agreements of Silicon in connection with the Exchange
                  Contracts, and without limiting any of the terms of such
                  applications and agreements, the Borrower will pay all
                  standard fees and charges of Silicon in connection with the
                  Exchange Contracts.

                  Without limiting any of the other terms of this Loan Agreement
                  or any such standard form applications and agreements of
                  Silicon, Borrower agrees to indemnify Silicon and hold it
                  harmless, from and against any and all claims, debts,
                  liabilities, demands, obligations, actions, costs and expenses
                  (including, without limitation, attorneys' fees of counsel of
                  Silicon's choice), of every nature and description, which it
                  may sustain or incur, based upon, arising out of, or in any
                  way relating to any of the Exchange Contracts or any
                  transactions relating thereto or contemplated thereby
                  (collectively referred to as the "FX Indemnity Provisions").

                  The Exchange Contracts shall have maturity dates no later than
                  the Maturity Date, provided maturities dates later than the
                  Maturity Date may be permitted in Silicon's discretion if, on
                  the Maturity Date, for any Exchange Contracts remaining
                  outstanding and having maturities after the Maturity Date,
                  Borrower provides to Silicon cash collateral in an amount
                  equal to the face amount of all such Exchange Contracts, plus
                  all interest, fees and costs due or to become due in
                  connection therewith, to secure all of the Obligations
                  relating to said Exchange Contracts, pursuant to Silicon's
                  then standard form cash pledge agreement.

                                      -3-
<PAGE>
 
Interest Rate
(Section 1.2):             A rate equal to the "Prime Rate" in effect from time
                           to time, calculated on the basis of a 360-day year
                           for the actual number of days elapsed. "Prime Rate"
                           means the rate announced from time to time by Silicon
                           as its "prime rate;" it is a base rate upon which
                           other rates charged by Silicon are based, and it is
                           not necessarily the best rate available at Silicon.
                           The interest rate applicable to the Obligations shall
                           change on each date there is a change in the Prime
                           Rate.

Loan Fee
(Section 1.3):             Per the Amendment to Loan Agreement of even date.

Maturity Date
(Section 5.1):             January 5, 1998.

Prior Names of Borrower
(Section 3.2):             None

Trade Names of Borrower
(Section 3.2):             Vitesse, VSC

Other Locations and 
Addresses (Section 3.3):   See Exhibit 1 hereto.

Material Adverse
Litigation (Section 3.10): None
                          
Negative Covenants-        Without Silicon's prior written consent, Borrower may
Exceptions (Section 4.6):  do the following, provided that, after giving effect
                           thereto, no Event of Default has occurred and no
                           event has occurred which, with notice or passage of
                           time or both, would constitute an Event of Default,
                           and provided that the following are done in
                           compliance with all applicable laws, rules and
                           regulations: (i) repurchase shares of Borrower's
                           stock pursuant to any employee stock purchase or
                           benefit plan, provided that the total amount paid by
                           Borrower for such stock does not exceed $500,000 in
                           any fiscal year or (ii) make loans to, or guarantee
                           indebtedness of, individuals in the usual and
                           ordinary course of business, provided that the
                           aggregate amount of any such loans or guaranties per
                           individual shall not exceed $200,000.

                     
                    
Financial Covenants        Borrower shall at the end of each quarter comply with
(Section 4.1):             all of the following covenants:

   Quick Ratio:            Borrower shall maintain a ratio of Quick Assets to
                           current liabilities of not less than 2.00 to 1.

                                      -4-
<PAGE>
 
Tangible Net        Borrower shall maintain a tangible net worth of not less
Worth:              than $70,000,000 plus 50% of all net income (after taxes)
                                     ---- 
                    earned in each fiscal quarter ending after the date of this
                    Agreement (in no event shall this number be considered to be
                    less than zero) plus 80% of the net proceeds received by the
                                    ----
                    Borrower of all equity offerings and other equity
                    transactions of the Borrower on and after September 30,
                    1996.

Debt to Tangible    Borrower shall maintain a ratio of total liabilities to
Net Worth Ratio:    tangible net worth of not more than .75 to 1.

Definitions:        "Current assets" shall have the meaning ascribed to it in
                    accordance with generally accepted accounting principles
                    ("GAAP").

                    "Current liabilities" shall have the meaning ascribed to it
                    in accordance with GAAP, except that deferred revenues shall
                    not be included as a current liability.

                    "Tangible net worth" means the excess of total assets over
                    total liabilities, determined in accordance with generally
                    accepted accounting principles, excluding however (i)
                    liabilities which may be converted to equity at the sole
                    discretion of the Borrower, and (ii) all assets which would
                    be classified as intangible assets under generally accepted
                    accounting principles, including without limitation
                    goodwill, licenses, patents, trademarks, trade names,
                    copyrights, and franchises.

                    "Quick Assets" means cash on hand or on deposit in banks,
                    readily marketable securities issued by the United States,
                    readily marketable commercial paper rated "A-l" by Standard
                    & Poor's Corporation (or a similar rating by a similar
                    rating organization), certificates of deposit and banker's
                    acceptances, and accounts receivable (net of allowance for
                    doubtful accounts).

Subordinated Debt:  "Liabilities" for purposes of the foregoing covenants do not
                    include (i) indebtedness which is subordinated to the
                    indebtedness to Silicon under a subordination agreement in
                    form specified by Silicon or by language in the instrument
                    evidencing the indebtedness which is acceptable to Silicon
                    or (ii) the indebtedness of the Borrower to Vitesse
                    Electronics Corporation, pursuant to the Asset Purchase
                    Agreement between Borrower and Vitesse Electronics
                    Corporation."

                                      -5-
<PAGE>
 
Other Covenants     Borrower shall at all times comply with all of the following
(Section 4.1):      additional covenants:

                    1. Banking Relationship. Borrower shall at all times
                    maintain its primary banking relationship with Silicon.

                    2. [Reserved]

                    3. Indebtedness. Without limiting any of the foregoing terms
                    or provisions of this Agreement, Borrower shall not in the
                    future incur indebtedness for borrowed money, except for (i)
                    indebtedness to Silicon, and (ii) financing relating to
                    purchase money or lease transaction in connection with the
                    Borrower's capital assets or lease of real estate assets.

                    4. Negative Pledge. Borrower shall not grant a security
                    interest in any of its Collateral, whether presently owned
                    or hereafter acquired, provided, however, Borrower may incur
                                           --------  -------  
                    liens on capital equipment and leases of real property
                    assets relating to the indebtedness incurred pursuant to
                    Paragraph 3 set forth above entitled "Indebtedness".

                    5. No Material Adverse Change; Event of Default. Without
                    limiting the Events of Default as set forth in the Loan
                    Agreement, any material adverse change in the business,
                    assets or prospects of Borrower after the date hereof shall
                    constitute an Event of Default hereunder.


                      Borrower:

                                 VITESSE SEMICONDUCTOR CORPORATION

                                 By /s/ [ILLEGIBLE SIGNATURE]
                                   -------------------------------
                                           Vice President

                                 By /s/ [ILLEGIBLE SIGNATURE]
                                   -------------------------------
                                           Ass't Secretary

                      Silicon:

                                 SILICON VALLEY BANK

                                 By /s/ [ILLEGIBLE SIGNATURE]
                                   -------------------------------
                                           Vice President


                                      -6-